WdsUpsertJobConfig
Creates or replaces the named job’s top-level crawler configuration, including start URLs, request settings, restart behavior, crawl scope, and retrieval indexing options
Arguments
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| jobName | string | Yes | None | Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
| jobConfig | JobConfig | Yes | None | Defines a job’s entry URLs and runtime options for downloading, crawling, scraping, retries, proxies, cross-domain links, and optional retrieval indexing |
Return Type
Returns void.
Technical return shape: void.
Data Contracts
JobConfig
Defines a job’s entry URLs and runtime options for downloading, crawling, scraping, retries, proxies, cross-domain links, and optional retrieval indexing
Fields:
| Name | Type | Description |
|---|---|---|
| StartUrls | array of string or null | Optional. Initial URLs used to create download tasks on job start. A saved config may omit them, but at least one URL is required to start the job |
| Type | JobType or null | Optional. Crawler network profile. Internet uses public-source request gateways; Intranet uses internal-source gateways. Omitted or null uses the first service-configured allowed job type |
| Headers | HeadersConfig or null | Optional. Additional HTTP headers for download requests. Omitted or null sends no additional default headers |
| Restart | JobRestartConfig or null | Optional. Behavior when starting a job that has prior tasks or cached data. Omitted or null uses Continue |
| Https | HttpsConfig or null | Optional. HTTPS certificate-validation behavior. Omitted or null validates certificates normally |
| Cookies | CookiesConfig or null | Optional. Cookie persistence behavior. Omitted or null does not persist cookies |
| Proxy | ProxiesConfig or null | Optional. Proxy routing, pool, and direct-request fallback behavior. Omitted or null uses non-proxy request gateways |
| DownloadErrorHandling | DownloadErrorHandlingConfig or null | Optional. Failed-download policy. Omitted or null does not retry failed requests |
| CrawlersProtectionBypass | CrawlersProtectionBypass or null | Optional. Response, timeout, redirect, and per-host pacing overrides. Omitted or null uses downloader defaults |
| CrossDomainAccess | CrossDomainAccessConfig or null | Optional. Discovered-link host policy. Omitted or null allows the current host and its subdomains |
| Retrieval | RetrievalConfig or null | Optional. Retrieval-index enrollment settings. Omitted or null does not enroll downloaded content |
| Host | HostConfig or null | Optional. Host lifecycle settings such as robots.txt and cookie refresh cadence. Omitted or null uses the 7-day reinitialization default |
JobType
Network profile used when the job is sent to the download service
Enumeration values:
| Name | Description |
|---|---|
| Internet | Use the public internet crawling profile, including request gateways such as proxies or host IP addresses when configured |
| Intranet | Use the intranet crawling profile for internal resources without public-source gateway restrictions |
Note: Dapi configuration can restrict the allowed values and set the default job type. Crawler must also be configured to handle every enabled job type.
HeadersConfig
Configures request headers applied to each download request
Fields:
| Name | Type | Description |
|---|---|---|
| DefaultRequestHeaders | array of HttpHeader or null | Optional. Additional HTTP headers sent with every request for this job. Omitted or empty sends no additional default headers |
HttpHeader
Represents one HTTP header with its name and one or more values
Fields:
| Name | Type | Description |
|---|---|---|
| Name | string | HTTP header name, for example User-Agent or Authorization |
| Values | array of string | One or more values for this header |
JobRestartConfig
Controls how StartJob behaves when the job has already been started before
Fields:
| Name | Type | Description |
|---|---|---|
| JobRestartMode | JobRestartModes | Optional. Controls whether a restarted job keeps previous task data or starts clean. Continue is the default and resumes from existing job data; FromScratch removes job tasks and related service data before crawling again from StartUrls. |
JobRestartModes
Allowed JobRestartMode values and their effect on existing task data
Enumeration values:
| Name | Description |
|---|---|
| Continue | Continue: keep existing task data and continue crawling or parsing remaining work |
| FromScratch | FromScratch: delete the job’s existing tasks and related service data, then crawl again from StartUrls |
HttpsConfig
Controls TLS certificate validation for HTTPS target resources
Fields:
| Name | Type | Description |
|---|---|---|
| SuppressHttpsCertificateValidation | bool | Optional. When true, accept invalid or self-signed HTTPS certificates for downloaded resources. False is the default |
CookiesConfig
Configures whether the downloader stores response cookies and sends them on later requests
Fields:
| Name | Type | Description |
|---|---|---|
| UseCookies | bool | Optional. When true, persist and reuse cookies between requests for this job. False is the default |
ProxiesConfig
Controls proxy usage, proxy pool entries, fallback to direct host requests, and response codes that trigger proxy rotation
Fields:
| Name | Type | Description |
|---|---|---|
| UseProxy | bool | Optional. When true, route download requests through the configured proxy pool. False is the default |
| SendOvertRequestsOnProxiesFailure | bool | Optional. When true, fall back to a direct non-proxy request after all eligible proxies fail. False is the default |
| Proxies | array of ProxyConfig or null | Optional. Proxy endpoints used when UseProxy is true. Null or empty provides no eligible proxies |
| IterateProxyResponseCodes | string or null | Optional. HTTP status codes that cause a retry through the next proxy, separated by commas, semicolons, pipes, or ampersands. Omitted or null uses 401 and 403 |
ProxyConfig
Defines one proxy endpoint, credentials, target-host allow list, connection limit, and renewal interval
Fields:
| Name | Type | Description |
|---|---|---|
| Protocol | string | Proxy protocol, for example http, https, or socks5 |
| Host | string | Proxy server host name or IP address |
| Port | int | Proxy server port |
| UserName | string or null | Optional. Proxy username |
| Password | string or null | Optional. Proxy password |
| ConnectionsLimit | int or null | Optional. Maximum concurrent connections through this proxy. Omitted or null applies no connection-count limit |
| AvailableHosts | array of string or null | Optional. Lowercase target hosts, or host:port values for non-default ports, that may use this proxy. Null or empty applies no target-host restriction |
| RenewalIntervalSec | int or null | Optional. Interval in seconds used to cap the wait before this proxy can be selected again for a target host |
DownloadErrorHandlingConfig
Controls how the crawler handles download failures before deciding whether the task is failed
Fields:
| Name | Type | Description |
|---|---|---|
| Policy | DownloadErrorHandlingPolicies | Policy determines what to do when a download request fails. Skip records the failed task and lets the crawl continue. Retry retries first and requires RetryPolicyParams with RetryDelayMs and RetriesLimit. |
| RetryPolicyParams | RetryPolicyParams or null | Optional. Required when Policy is Retry and otherwise unused. Specifies the delay and maximum retry count |
DownloadErrorHandlingPolicies
Available strategies for failed download requests
Enumeration values:
| Name | Description |
|---|---|
| Skip | Record the failed request and continue crawling without retrying |
| Retry | Retry the failed request according to RetryPolicyParams before marking it failed |
RetryPolicyParams
Configures retry delay and maximum retry attempts for failed downloads
Fields:
| Name | Type | Description |
|---|---|---|
| RetryDelayMs | int | Delay in milliseconds before each retry attempt |
| RetriesLimit | int | Maximum number of retry attempts |
CrawlersProtectionBypass
Configures per-download response, redirect, timeout, and per-host pacing limits
Fields:
| Name | Type | Description |
|---|---|---|
| MaxResponseSizeKb | int or null | Optional. Maximum response content buffered for one download, in kilobytes. Omitted or null uses 1000 KB |
| MaxRedirectHops | int or null | Optional. Maximum number of redirects followed for one download. Omitted or null uses 10; 0 disables redirects |
| RequestTimeoutSec | int or null | Optional. Request timeout in seconds. Omitted or null uses 30 seconds |
| CrawlDelays | array of CrawlDelay or null | Optional. Per-host delay overrides. Omitted or empty uses the robots.txt crawl-delay when available, otherwise no extra delay |
CrawlDelay
Configures request pacing for one target host
Fields:
| Name | Type | Description |
|---|---|---|
| Host | string | Lowercase target host, or host:port for a non-default port, to which this delay applies |
| Delay | string | Per-host wait between requests. Use ‘0’ for no extra delay, an integer for fixed seconds, a range such as ‘1-5’ for a random value from 1 inclusive to 5 exclusive, or ‘robots’ to use the robots.txt crawl-delay when available |
CrossDomainAccessConfig
Controls whether the crawler follows links whose host differs from the current page host
Fields:
| Name | Type | Description |
|---|---|---|
| Policy | CrossDomainAccessPolicies | Optional. Selects which discovered links may be followed. Set this explicitly when providing CrossDomainAccessConfig; the enum default is None. Omitting the entire CrossDomainAccess config uses Subdomains |
CrossDomainAccessPolicies
Domain scoping modes that determine which discovered links are in bounds
Enumeration values:
| Name | Description |
|---|---|
| None | Follow only links on the same host as the current page |
| Subdomains | Follow links on the same host and its subdomains |
| CrossDomains | Follow links to any host |
RetrievalConfig
Configures optional enrollment of downloaded page content into the retrieval index
Fields:
| Name | Type | Description |
|---|---|---|
| EnrollInIndex | bool | Optional. When true, convert successfully downloaded page content to markdown chunks and enroll it for retrieval. False is the default. Cached pages are skipped unless Force is true |
| Force | bool | Optional. When true, enroll cached page content instead of skipping it. Has no effect when EnrollInIndex is false |
| MaxTokensPerChunk | int or null | Optional. Maximum tokens per indexed chunk. Omitted or null uses 512 |
| ContentScopes | array of RetrievalContentScope or null | Optional. Ordered URL-path patterns and selectors for enrolled content. The first matching scope supplies the selector; if none matches, the whole page is enrolled |
| WaitForEnrolled | bool | Optional. When true, wait up to one minute per page for its snippets to become queryable before continuing. False is the default and returns after submitting enrollment |
RetrievalContentScope
Defines which part of matching pages is enrolled by pairing a URL path glob with a content selector
Fields:
| Name | Type | Description |
|---|---|---|
| PathPattern | string | Case-sensitive URL path glob. Use ‘’ for one path segment or file-name part and ‘’ for nested folders. Example: /products//details/.html |
| Selector | string | Selector for the page content to convert to markdown and enroll when PathPattern matches |
Path pattern examples:
| URL | Pattern | Matches |
|---|---|---|
https://example.com/path/to/resource |
* |
Yes |
https://example.com/path/to/resource |
/* |
Yes |
https://example.com/path/to/resource |
/path/to/resource |
Yes |
https://example.com/path/to/resource |
/path/to/* |
Yes |
https://example.com/path/to/resource |
/path/*/resource |
Yes |
https://example.com/path/to/resource |
/**/res* |
Yes |
https://example.com/path/to/resource |
/res* |
No |
https://example.com/path/to/resource |
/path/to/RESOURCE |
No |
HostConfig
Configures host-level refresh behavior used by the downloader
Fields:
| Name | Type | Description |
|---|---|---|
| ReinitPeriodSec | int or null | Optional. Host reinitialization interval in seconds. Reinitialization refreshes robots.txt content and renews cookies. Omitted or null uses 604800 seconds (7 days) |
Behavior
Read-only: no; idempotent: yes; destructive: yes; open-world: no.