JobConfig
Job configuration
Fields
| Name | Type | Description |
|---|---|---|
| Server | ServerConfig | Required. WDS API Server connection parameters. |
| StartUrls | Array of Strings | Required. Initial URLs. Craling entry points |
| JobName | String | Optional. Job name. If not specified a random generated value is used. |
| JobType | JobTypes | Optional. Job type |
| JobRestart | JobRestartConfig | Optional. Job restart settings |
| Https | HttpsConfig | Optional. HTTPS settings |
| Cookies | CookiesConfig | Optional. Cookies settings |
| Proxy | ProxiesConfig | Optional. Proxy settings |
| DownloadErrorHandlingPolicy | DownloadErrorHandlingPolicy | Optional. Download errors handling settings |
| CrawlersProtectionBypass | CrawlersProtectionBypass | Optional. Crawlers protection countermeasure settings |
Initialization String Format
An instance can be initialized with a string of the following format: JobName: jobname; Server: serverConnectionString; StartUrls: URL1, URL2
Methods
Methods that help with initialization.
AddStartUrl
Adds a new start URL
Syntax
AddStartUrl( url )
Arguments
| Name | Type | Description |
|---|---|---|
| url | String | Start Url |
Return type
Return value
Returns the instance on which it was called
Examples
Creating a new instance initialized from a string:
DECLARE @jobConfig wds.JobConfig = 'JobName: TestJob1; ServerCS: wds://localhost:2807; StartUrls: http://playground.svc';
Adding one more start URL:
SET @jobConfig = @jobConfig.AddStartUrl('http://example.com');
JobTypes
Job types enumeration.
Possible values restrictions and the default value for all jobs can be configured in the Dapi service.
Additionally, the Crawler service should be correctly configured to handle jobs of different types.
Values
| Name | Description |
|---|---|
| Internet | Crawl data from internet sources via request gateways (Proxy addresses, Host IP addresses, etc.) |
| Intranet | Crawl data from intranet sources with no limits |
Examples
Changing the job type:
SET @jobConfig.JobType = 'Intranet';