Start

Starts or restarts the job named in the supplied configuration after applying its crawler settings, then returns its initial download tasks.

Syntax

wds.Start(@jobConfig)

Arguments

Name	Type	Description
@jobConfig	wds.JobConfig	Supplies the server connection, job name, start URLs, and crawler settings. Its job name selects the MSSQL-compatible job to configure and start.

Returns

TABLE (Task wds.DownloadTask) — wds.DownloadTask values, one for each configured start URL.

Data Contracts

JobConfig

Configures a WDS job and its request, crawl, retrieval, and host behavior.

Name	Type	Description
Server	ServerConfig	WDS API server connection parameters.
JobType	string or null	String wrapper for `Type`; excluded from XML serialization.
JobName	string	Optional job name; when not specified, a random value is generated.
StartUrls	array of string	Initial URLs that start the job.
Type	JobTypes	Optional job type.
Headers	HeadersConfig or null	Optional headers settings.
Restart	RestartConfig or null	Optional job restart settings.
Https	HttpsConfig or null	Optional HTTPS settings.
Cookies	CookiesConfig or null	Optional cookies settings.
Proxy	ProxiesConfig or null	Optional proxy settings.
DownloadErrorHandling	DownloadErrorHandling or null	Optional download-error handling settings.
CrawlersProtectionBypass	CrawlersProtectionBypass or null	Optional crawler-protection countermeasure settings.
CrossDomainAccess	CrossDomainAccess or null	Optional cross-domain access settings.
Retrieval	RetrievalConfig or null	Optional retrieval-index enrollment settings.
Host	HostConfig or null	Optional host lifecycle settings.

AddStartUrl(url) adds an initial URL. Parse accepts the semicolon-separated job configuration; validation requires server and start URLs, then validates configured nested contracts.

ServerConfig

API server configuration.

Name	Type	Description
Uri	Uri	API server URI.

Parse accepts wds://user:password@host:port?https=false, requires the wds scheme, and produces an HTTP URI unless https=true is supplied. Validation requires a URI.

JobTypes

Defines the environment in which a job crawls.

Name	Description
Internet	Crawls internet sources using request gateways.
Intranet	Crawls intranet sources without those limits.

HeadersConfig

Configures HTTP headers applied to all job requests.

Name	Type	Description
DefaultRequestHeaders	array of HttpHeader or null	Optional headers sent with all requests.

AddHeader adds a header; AppendHeader adds a value to a named header. Parse accepts the configured header fields.

HttpHeader

HTTP header.

Name	Type	Description
Name	string	Header name.
Values	array of string	Header values.

Parse accepts Name: name; Values: value1, value2;. Validation requires both fields.

RestartConfig

Configures how an existing job is restarted.

Name	Type	Description
RestartMode	string	SQL-facing string representation of `JobRestartMode`.
JobRestartMode	JobRestartModes	Restart mode.

Parse accepts RestartMode: Continue|FromScratch.

JobRestartModes

Defines the behavior of a restarted job.

Name	Description
Continue	Reuses cached data and continues crawling and parsing new data.
FromScratch	Clears cached data and starts from scratch.

HttpsConfig

Configures HTTPS certificate validation for target web resources.

Name	Type	Description
SuppressHttpsCertificateValidation	bool	When true, suppresses HTTPS certificate validation.

Parse accepts SuppressHttpsCertificateValidation: true|false.

CookiesConfig

Configures whether the downloader stores response cookies and sends them on later requests.

Name	Type	Description
UseCookies	bool	When true, persists and reuses cookies between requests for the job; false is the default.

Parse accepts UseCookies: true|false.

ProxiesConfig

Configures whether requests use proxies and the pool of available proxy endpoints.

Name	Type	Description
UseProxy	bool	Enables proxy use for requests.
SendOvertRequestsOnProxiesFailure	bool	Enables direct requests when all proxies fail.
IterateProxyResponseCodes	string	Optional comma-separated HTTP status codes that cause proxy rotation.
Proxies	array of ProxyConfig or null	Optional proxy configurations.

AddProxyConfig adds a proxy configuration. AddProxy creates and adds one from its protocol, host, port, optional credentials, connection limit, and allowed hosts.

ProxyConfig

Configures one proxy endpoint for download requests.

Name	Type	Description
Protocol	string	Proxy protocol (`http`, `https`, or `socks5`).
Host	string	Proxy host.
Port	int	Proxy port.
UserName	string or null	Optional proxy username.
Password	string or null	Optional proxy password.
ConnectionsLimit	int or null	Optional maximum concurrent connections through this proxy; null applies no limit.
AvailableHosts	array of string or null	Optional lowercase target hosts (or `host:port`) that may use this proxy; null or empty applies no restriction.
RenewalIntervalSec	int or null	Optional interval in seconds that caps the wait before this proxy can again be selected for a target host.

Parse accepts the semicolon-separated fields shown above. Validation requires Host and a nonzero Port. AddAvailableHost(host) adds a distinct target host.

DownloadErrorHandling

Configures the policy applied when a download fails.

Name	Type	Description
ErrorHandlingPolicy	string	SQL-facing string representation of `Policy`; excluded from XML serialization.
Policy	DownloadErrorHandlingPolicies	Error-handling policy.
RetryPolicyParams	RetryPolicyParams or null	Optional retry settings, used when the policy is `Retry`.

Parse accepts ErrorHandlingPolicy: Skip|Retry.

DownloadErrorHandlingPolicies

Defines how the downloader reacts to an error.

Name	Description
Skip	Skips an error and continues crawling.
Retry	Retries according to the configured retry policy.

RetryPolicyParams

Retry settings.

Name	Type	Description
RetryDelayMs	int	Delay between retries in milliseconds.
RetriesLimit	int	Maximum number of retries.

Parse accepts RetryDelayMs: 1000; RetriesLimit: 3.

CrawlersProtectionBypass

Configures per-download response, redirect, timeout, and per-host pacing limits.

Name	Type	Description
MaxResponseSizeKb	int or null	Optional maximum response content buffered for a download; null uses 1000 KB.
MaxRedirectHops	int or null	Optional maximum redirects; null uses 10 and `0` disables redirects.
RequestTimeoutSec	int or null	Optional request timeout in seconds; null uses 30 seconds.
CrawlDelays	array of CrawlDelay or null	Optional per-host delay overrides; null or empty uses robots.txt delay when available, otherwise no extra delay.

AddCrawlDelay adds or replaces a host delay. AddDelay creates a delay from a host and delay string. Validation applies to all configured delays.

CrawlDelay

Configures request pacing for one target host.

Name	Type	Description
Host	string	Lowercase target host, or `host:port` for a non-default port, to which the delay applies.
Delay	string	Per-host request wait: `0`, fixed seconds, a range such as `1-5`, or `robots` to use the robots.txt crawl-delay when available.

Parse accepts Host: host.com; Delay: 0|1-5|robots. Validation requires both fields and a supported delay format.

CrossDomainAccess

Configures cross-domain navigation behavior for a job.

Name	Type	Description
AccessPolicy	CrossDomainAccessPolicies	Cross-domain access policy.

Parse accepts AccessPolicy: None|Subdomains|CrossDomains. Validation requires a defined access policy.

CrossDomainAccessPolicies

Defines the permitted navigation scope for a crawl.

Name	Description
None	Allows only the original domain.
Subdomains	Allows the original domain and its subdomains.
CrossDomains	Allows navigation to any domain.

RetrievalConfig

Configures optional enrollment of downloaded page content into the retrieval index.

Name	Type	Description
EnrollInIndex	bool	When true, converts successfully downloaded page content to Markdown chunks and enrolls it for retrieval. False is the default; cached pages are skipped unless `Force` is true.
Force	bool	When true, enrolls cached page content instead of skipping it; it has no effect when `EnrollInIndex` is false.
MaxTokensPerChunk	int or null	Optional maximum tokens per indexed chunk; null uses 512.
ContentScopes	array of RetrievalContentScope or null	Optional ordered URL-path patterns and selectors; the first matching scope supplies the selector, otherwise the whole page is enrolled.
WaitForEnrolled	bool	When true, waits up to one minute per page for snippets to become queryable; false returns after submitting enrollment.

AddContentScope appends a scope; the method has no XML summary in the source. Validation validates every configured scope.

RetrievalContentScope

Indexed content scope.

Name	Type	Description
PathPattern	string	Source XML documentation is missing for this public member.
Selector	string	Source XML documentation is missing for this public member.

Validation requires both fields. The source needs XML summaries for PathPattern and Selector before their behavior can be documented more specifically.

HostConfig

Configures host lifecycle settings such as robots.txt and cookie refresh cadence.

Name	Type	Description
ReinitPeriodSec	int or null	Optional host reinitialization interval in seconds; null uses 604800 seconds (7 days). Reinitialization refreshes robots.txt content and renews cookies.

Parse accepts ReinitPeriodSec: seconds.

DownloadTask

Represents a download task that can be passed to MSSQL CLR functions.

Name	Type	Description
Server	ServerConfig	Server connection associated with the task.
Id	string	Download task identifier.
Url	string	Download task URL.

Validation requires Server, Id, and Url.