# Release v3.0 # WDS API Server # API # API The WDS API exposes versioned endpoints for stored crawler jobs, task-level crawl and scrape processing, Traversal runs and schedules, retrieval-index search, and tenant cleanup. The base URL depends on the [deployment option](../deployments/index.html); a local Docker deployment normally exposes Dapi at `http://localhost:2807`. All routes below use `/api/v3`. The core routes require the `Api` feature, and schedule routes require the `Traversal Scheduling` feature. Scheduling is available starting with the Business plan. See [Plans](https://webdatasource.com/pricing.html) for feature availability. Swagger UI is available at `/api/swagger`. In debug builds, `X-Tenant-Id` can select a tenant; production uses the configured default tenant. ## Jobs | Operation | Route | Description | | --- | --- | --- | | [Get Jobs Info](../api/jobs.html#get-jobs-info) | `GET /jobs` | Lists jobs in the current tenant for discovery and lifecycle monitoring | | [Upsert a Job](../api/jobs.html#upsert-a-job) | `POST /jobs/{jobName}/config` | Creates or replaces the named job's top-level crawler configuration, including start URLs, request settings, restart behavior, crawl scope, and retrieval indexing options | | [Get Config](../api/jobs.html#get-config) | `GET /jobs/{jobName}/config` | Reads the named job's current crawler configuration before inspection or replacement | | [Start a Job](../api/jobs.html#start-a-job) | `POST /jobs/{jobName}/start` | Starts the named job from its configured start URLs. If the job was started before, applies its restart policy, then returns the initial download task IDs | | [Delete Job](../api/jobs.html#delete-job) | `DELETE /jobs/{jobName}` | Deletes the named job and cleans up its Traversal config, schedule, tasks, run history, downloaded and scraped data, retrieval-index data, and service-side settings for the current tenant | ## Processing | Operation | Route | Description | | --- | --- | --- | | [Fetch](../api/processing.html#fetch) | `POST /jobs/{jobName}/fetch` | Downloads one absolute URL with the named job's crawler settings and returns page content when ready, or a task content URL while pending | | [Crawl](../api/processing.html#crawl) | `POST /tasks/{taskId}/crawl` | Extracts links from the downloaded page using CrawlParams, applies job scope and maxDepth rules, creates child download tasks, and completes the current task. If processing is pending, poll the crawl-result URL | | [Crawl Result](../api/processing.html#crawl-result) | `GET /tasks/{taskId}/crawl-result` | Polls or resumes a previous Crawl request for this task using its stored CrawlParams until processing completes | | [Scrape](../api/processing.html#scrape) | `POST /tasks/{taskId}/scrape` | Extracts named values from the downloaded page using ScrapeParams selectors and optional conversions, then completes the current task. If processing is pending, poll the scrape-result URL | | [Scrape Result](../api/processing.html#scrape-result) | `GET /tasks/{taskId}/scrape-result` | Polls or resumes a previous Scrape request for this task using its stored ScrapeParams until processing completes | | [Content](../api/processing.html#content) | `GET /tasks/{taskId}/content` | Reads all or part of the downloaded page for a task, typically one created by Fetch. If processing is pending, poll the same content URL | | [Download Status](../api/processing.html#download-status) | `GET /tasks/{taskId}/download-status` | Inspects a task's download state and HTTP attempt history to troubleshoot pending, failed, or completed downloads | ## Retrieval Retrieval routes require the `Retrieval` feature. See [Plans](https://webdatasource.com/pricing.html) for feature availability. | Operation | Route | Description | | --- | --- | --- | | [Tenant-wide Search](../api/retrieval.html#tenant-wide-search) | `POST /jobs/retrieve` | Searches retrieval-indexed crawled content across all jobs in the current tenant to find evidence relevant to a question | | [Job-scoped Search](../api/retrieval.html#job-scoped-search) | `POST /jobs/{jobName}/retrieve` | Searches retrieval-indexed crawled content produced by one named job to find evidence relevant to a question | ## Traversal | Operation | Route | Description | | --- | --- | --- | | [Upsert Traversal Config](../api/traversal.html#upsert-traversal-config) | `POST /jobs/{jobName}/traversal/config` | Validates and stores the named job's TraversalConfig multi-level crawl and scrape plan in the Traversal runner. This does not start a run | | [Get Traversal Config](../api/traversal.html#get-traversal-config) | `GET /jobs/{jobName}/traversal/config` | Reads the named job's saved TraversalConfig so its multi-level crawl and scrape plan can be inspected or reused | | [Delete Traversal](../api/traversal.html#delete-traversal) | `DELETE /jobs/{jobName}/traversal` | Deletes the named job's saved TraversalConfig, schedule, tasks, and run history from the Traversal runner. This does not delete the job | | [Start Traversal Run](../api/traversal.html#start-traversal-run) | `POST /jobs/{jobName}/traversal/start` | Requests a Traversal run using the named job's current job config and TraversalConfig, reusing an active run instead of starting a duplicate. Use the returned run number for later inspection | | [List Traversal Runs](../api/traversal.html#list-traversal-runs) | `GET /jobs/{jobName}/traversal/runs` | Lists recent Traversal run summaries for the named job, newest first, so clients can find run numbers and completion state | | [Get Traversal Run Info](../api/traversal.html#get-traversal-run-info) | `GET /jobs/{jobName}/traversal/runs/{runNum}/info` | Inspects the progress and configuration used by one Traversal run | | [Get Traversal Run Errors](../api/traversal.html#get-traversal-run-errors) | `GET /jobs/{jobName}/traversal/runs/{runNum}/errors` | Investigates failed downloads from one Traversal run and the parent pages that discovered them | | [Get Traversal Run Data](../api/traversal.html#get-traversal-run-data) | `GET /jobs/{jobName}/traversal/runs/{runNum}/data` | Reads a batch of scraped data produced by a Traversal run. Continue with the returned cursor until no cursor remains | ## Traversal Scheduling | Operation | Route | Description | | --- | --- | --- | | [Upsert Traversal Schedule](../api/traversal-scheduling.html#upsert-traversal-schedule) | `POST /jobs/{jobName}/traversal/schedule` | Creates or replaces the named job's TraversalSchedule for automatic runs. A saved TraversalConfig is required. This configures future runs and does not start one immediately | | [Get Traversal Schedule](../api/traversal-scheduling.html#get-traversal-schedule) | `GET /jobs/{jobName}/traversal/schedule` | Reads the named job's saved automatic Traversal schedule. A saved TraversalConfig is required | | [Delete Traversal Schedule](../api/traversal-scheduling.html#delete-traversal-schedule) | `DELETE /jobs/{jobName}/traversal/schedule` | Deletes the named job's TraversalSchedule and stops future scheduled starts. A saved TraversalConfig is required. This does not delete the job, Traversal config, or previous runs | ## Tenants | Operation | Route | Description | | --- | --- | --- | | [Delete Tenant](../api/tenants.html#delete-tenant) | `DELETE /tenants/{tenantId}` | Deletes a tenant and cleans up its jobs, download tasks, Traversal configs, schedules, tasks and run history, cached and scraped data, and retrieval-index data across backing services | ## Typical Flow 1. [Upsert a job](../api/jobs.html#upsert-a-job) with its reusable `JobConfig`. 2. [Start the job](../api/jobs.html#start-a-job) and collect its initial task IDs. 3. [Crawl](../api/processing.html#crawl), [scrape](../api/processing.html#scrape), or [read content](../api/processing.html#content) from those tasks. 4. When a task operation returns 202 (Accepted), retry the URL in `Location` after `Retry-After` until the result is ready. 5. For a multi-level plan, save a [Traversal config](../api/traversal.html#upsert-traversal-config), [start a run](../api/traversal.html#start-traversal-run), and page through its [run data](../api/traversal.html#get-traversal-run-data). # Quickstart — up and running in a couple of minutes Deploy WDS and run your first crawl/scrape entirely in Swagger UI. ## Prerequisites - Docker installed and running - WDS deployed following the guide: [Deploying WDS API Server in Docker Compose](../deployments/dockercompose.html) using the reconciled `MINI (Free)` option, with its MongoDB connection and embedding-service URL placeholders replaced Once deployed, the API is available at: `http://localhost:2807` ## Step 1 — Open Swagger UI Open: `http://localhost:2807/api/swagger` You’ll use six endpoints: - Jobs -> POST `{jobName}/config` — create or update the job configuration - Jobs -> POST `{jobName}/start` — start the stored job configuration and get the initial task IDs - Tasks -> POST `{taskId}/crawl` — discover follow-up pages (links) from a page - Tasks -> GET `{taskId}/crawl-result` — poll for pending crawl results - Tasks -> POST `{taskId}/scrape` — extract data from a page - Tasks -> GET `{taskId}/scrape-result` — poll for pending scrape results ## Step 2 — Create a job configuration In Swagger UI: 1. Expand Jobs -> POST `{jobName}/config`, then click “Try it out”. 2. Path parameter `jobName`: enter `playground` (or any unique name). 3. Request body: ```json { "startUrls": ["http://playground"], "type": "Intranet" } ``` 4. Click “Execute”. Response: `202 Accepted` confirms the job configuration was saved. ## Step 3 — Start the job In Swagger UI: 1. Expand Jobs -> POST `{jobName}/start`, then click “Try it out”. 2. Path parameter `jobName`: enter the same job name from Step 2, for example `playground`. 3. Click “Execute”. Response: `200 OK` returns an array of download task IDs. Copy one value; this is your first page task ID. ## Step 4 — Discover pages (Crawl) In Swagger UI: 1. Expand Tasks -> POST `{taskId}/crawl`, then click “Try it out”. 2. Path parameter `taskId`: paste the task ID from the Start response. 3. Request body: ```json { "selector": "css: a[href*='/cloak_of_the_phantom.html']", "attributeName": "href" } ``` 4. Click “Execute”. If the response is `202 Accepted`, wait for the `Retry-After` interval and call the URL in the `Location` header (Tasks -> GET `{taskId}/crawl-result`). Repeat until it returns `200 OK`. The `200 OK` response contains an array of new DownloadTask items (in this example, a single item). Copy its `id` value for the scraping step. ## Step 5 — Extract content (Scrape) In Swagger UI: 1. Expand Tasks -> POST `{taskId}/scrape`, then click “Try it out”. 2. Path parameter `taskId`: paste the selected task id from Step 4. 3. Request body: ```json [ { "name": "Title", "selector": "css: h1" }, { "name": "Price", "selector": "css: div.price span" }, { "name": "Description", "selector": "css: div.desc p" } ] ``` 4. Click “Execute”. If the response is `202 Accepted`, wait for the `Retry-After` interval and call the URL in the `Location` header (Tasks -> GET `{taskId}/scrape-result`). Repeat until it returns `200 OK`. The `200 OK` response contains an array of objects with values for each field. For example: ```json [ { "name": "Title", "values": [ "Cloak of the Phantom" ] }, { "name": "Price", "values": [ "100 Fairy Coins" ] }, { "name": "Description", "values": [ "Made from the feathers of a phoenix, it grants the power of rebirth." ] } ] ``` You’ve successfully extracted data — all within Swagger UI. ## Conclusion That’s it — deploy, configure, start, crawl, and scrape using only Swagger UI. For more, see the full [API docs](./index.html) and [Services](../services/index.html). # Jobs ## Get Jobs Info Lists jobs in the current tenant for discovery and lifecycle monitoring ### Endpoint `GET /api/v3/jobs` ### Responses #### 200 (Ok) Returns `JobInfo[]` as an array of [JobInfo](#jobinfo). #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Upsert a Job Creates or replaces the named job's top-level crawler configuration, including start URLs, request settings, restart behavior, crawl scope, and retrieval indexing options ### Endpoint `POST /api/v3/jobs/{jobName}/config` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Request Body Required `application/json` body containing a [JobConfig](#jobconfig). ### Responses #### 204 (No Content) Returns `void` with no response body. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Get Config Reads the named job's current crawler configuration before inspection or replacement ### Endpoint `GET /api/v3/jobs/{jobName}/config` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Responses #### 200 (Ok) Returns `JobConfig` as a [JobConfig](#jobconfig). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information #### 404 (Not Found) Job not found ## Start a Job Starts the named job from its configured start URLs. If the job was started before, applies its restart policy, then returns the initial download task IDs ### Endpoint `POST /api/v3/jobs/{jobName}/start` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Responses #### 200 (Ok) Returns `string[]`, an array of download task IDs. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information #### 404 (Not Found) Job not found ## Delete Job Deletes the named job and cleans up its Traversal config, schedule, tasks, run history, downloaded and scraped data, retrieval-index data, and service-side settings for the current tenant ### Endpoint `DELETE /api/v3/jobs/{jobName}` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Responses #### 204 (No Content) Returns `void` with no response body. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information #### 404 (Not Found) Job not found ## Data Contracts ### JobInfo Summary information for a configured job Fields: | Name | Type | Description | | --- | --- | --- | | JobId | string | Job ID | | JobName | string | Job name unique within the tenant | | StartDateUtc | datetime | Optional. Date and time when the job was last started, in UTC; null if it has not started | | CompleteDateUtc | datetime | Optional. Date and time when the job completed, in UTC; null if it has not completed | | EnrollmentState | [EnrollmentState](#enrollmentstate) | Optional. Current retrieval enrollment state; null when the retrieval service is unavailable | ### EnrollmentState Retrieval enrollment lifecycle state for indexed content produced by the job Enumeration values: | Name | Description | | --- | --- | | Pending | No indexed snippets have been observed for the job yet | | Enrolling | At least one indexed snippet is still being enrolled | | Completed | Indexed snippets for the job are visible in the retrieval index | | Disabled | Retrieval enrollment is disabled for the job | ### JobConfig Defines a job's entry URLs and runtime options for downloading, crawling, scraping, retries, proxies, cross-domain links, and optional retrieval indexing Fields: | Name | Type | Description | | --- | --- | --- | | StartUrls | array of string | Optional. Initial URLs used to create download tasks on job start. A saved config may omit them, but at least one URL is required to start the job | | Type | [JobType](#jobtype) | Optional. Crawler network profile. Internet uses public-source request gateways; Intranet uses internal-source gateways. Omitted or null uses the first service-configured allowed job type | | Headers | [HeadersConfig](#headersconfig) | Optional. Additional HTTP headers for download requests. Omitted or null sends no additional default headers | | Restart | [JobRestartConfig](#jobrestartconfig) | Optional. Behavior when starting a job that has prior tasks or cached data. Omitted or null uses Continue | | Https | [HttpsConfig](#httpsconfig) | Optional. HTTPS certificate-validation behavior. Omitted or null validates certificates normally | | Cookies | [CookiesConfig](#cookiesconfig) | Optional. Cookie persistence behavior. Omitted or null does not persist cookies | | Proxy | [ProxiesConfig](#proxiesconfig) | Optional. Proxy routing, pool, and direct-request fallback behavior. Omitted or null uses non-proxy request gateways | | DownloadErrorHandling | [DownloadErrorHandlingConfig](#downloaderrorhandlingconfig) | Optional. Failed-download policy. Omitted or null does not retry failed requests | | CrawlersProtectionBypass | [CrawlersProtectionBypass](#crawlersprotectionbypass) | Optional. Response, timeout, redirect, and per-host pacing overrides. Omitted or null uses downloader defaults | | CrossDomainAccess | [CrossDomainAccessConfig](#crossdomainaccessconfig) | Optional. Discovered-link host policy. Omitted or null allows the current host and its subdomains | | Retrieval | [RetrievalConfig](#retrievalconfig) | Optional. Retrieval-index enrollment settings. Omitted or null does not enroll downloaded content | | Host | [HostConfig](#hostconfig) | Optional. Host lifecycle settings such as robots.txt and cookie refresh cadence. Omitted or null uses the 7-day reinitialization default | ### JobType Network profile used when the job is sent to the download service Enumeration values: | Name | Description | | --- | --- | | Internet | Use the public internet crawling profile, including request gateways such as proxies or host IP addresses when configured | | Intranet | Use the intranet crawling profile for internal resources without public-source gateway restrictions | > **Note:** Dapi configuration can restrict the allowed values and set the default job type. Crawler must also be configured to handle every enabled job type. ### HeadersConfig Configures request headers applied to each download request Fields: | Name | Type | Description | | --- | --- | --- | | DefaultRequestHeaders | array of [HttpHeader](#httpheader) | Optional. Additional HTTP headers sent with every request for this job. Omitted or empty sends no additional default headers | ### HttpHeader Represents one HTTP header with its name and one or more values Fields: | Name | Type | Description | | --- | --- | --- | | Name | string | HTTP header name, for example User-Agent or Authorization | | Values | array of string | One or more values for this header | ### JobRestartConfig Controls how StartJob behaves when the job has already been started before Fields: | Name | Type | Description | | --- | --- | --- | | JobRestartMode | [JobRestartModes](#jobrestartmodes) | Optional. Controls whether a restarted job keeps previous task data or starts clean. Continue is the default and resumes from existing job data; FromScratch removes job tasks and related service data before crawling again from StartUrls. | ### JobRestartModes Allowed JobRestartMode values and their effect on existing task data Enumeration values: | Name | Description | | --- | --- | | Continue | Continue: keep existing task data and continue crawling or parsing remaining work | | FromScratch | FromScratch: delete the job's existing tasks and related service data, then crawl again from StartUrls | ### HttpsConfig Controls TLS certificate validation for HTTPS target resources Fields: | Name | Type | Description | | --- | --- | --- | | SuppressHttpsCertificateValidation | bool | Optional. When true, accept invalid or self-signed HTTPS certificates for downloaded resources. False is the default | ### CookiesConfig Configures whether the downloader stores response cookies and sends them on later requests Fields: | Name | Type | Description | | --- | --- | --- | | UseCookies | bool | Optional. When true, persist and reuse cookies between requests for this job. False is the default | ### ProxiesConfig Controls proxy usage, proxy pool entries, fallback to direct host requests, and response codes that trigger proxy rotation Fields: | Name | Type | Description | | --- | --- | --- | | UseProxy | bool | Optional. When true, route download requests through the configured proxy pool. False is the default | | SendOvertRequestsOnProxiesFailure | bool | Optional. When true, fall back to a direct non-proxy request after all eligible proxies fail. False is the default | | Proxies | array of [ProxyConfig](#proxyconfig) | Optional. Proxy endpoints used when UseProxy is true. Null or empty provides no eligible proxies | | IterateProxyResponseCodes | string | Optional. HTTP status codes that cause a retry through the next proxy, separated by commas, semicolons, pipes, or ampersands. Omitted or null uses 401 and 403 | ### ProxyConfig Defines one proxy endpoint, credentials, target-host allow list, connection limit, and renewal interval Fields: | Name | Type | Description | | --- | --- | --- | | Protocol | string | Proxy protocol, for example http, https, or socks5 | | Host | string | Proxy server host name or IP address | | Port | int | Proxy server port | | UserName | string | Optional. Proxy username | | Password | string | Optional. Proxy password | | ConnectionsLimit | int | Optional. Maximum concurrent connections through this proxy. Omitted or null applies no connection-count limit | | AvailableHosts | array of string | Optional. Lowercase target hosts, or host:port values for non-default ports, that may use this proxy. Null or empty applies no target-host restriction | | RenewalIntervalSec | int | Optional. Interval in seconds used to cap the wait before this proxy can be selected again for a target host | ### DownloadErrorHandlingConfig Controls how the crawler handles download failures before deciding whether the task is failed Fields: | Name | Type | Description | | --- | --- | --- | | Policy | [DownloadErrorHandlingPolicies](#downloaderrorhandlingpolicies) | Policy determines what to do when a download request fails. Skip records the failed task and lets the crawl continue. Retry retries first and requires RetryPolicyParams with RetryDelayMs and RetriesLimit. | | RetryPolicyParams | [RetryPolicyParams](#retrypolicyparams) | Optional. Required when Policy is Retry and otherwise unused. Specifies the delay and maximum retry count | ### DownloadErrorHandlingPolicies Available strategies for failed download requests Enumeration values: | Name | Description | | --- | --- | | Skip | Record the failed request and continue crawling without retrying | | Retry | Retry the failed request according to RetryPolicyParams before marking it failed | ### RetryPolicyParams Configures retry delay and maximum retry attempts for failed downloads Fields: | Name | Type | Description | | --- | --- | --- | | RetryDelayMs | int | Delay in milliseconds before each retry attempt | | RetriesLimit | int | Maximum number of retry attempts | ### CrawlersProtectionBypass Configures per-download response, redirect, timeout, and per-host pacing limits Fields: | Name | Type | Description | | --- | --- | --- | | MaxResponseSizeKb | int | Optional. Maximum response content buffered for one download, in kilobytes. Omitted or null uses 1000 KB | | MaxRedirectHops | int | Optional. Maximum number of redirects followed for one download. Omitted or null uses 10; 0 disables redirects | | RequestTimeoutSec | int | Optional. Request timeout in seconds. Omitted or null uses 30 seconds | | CrawlDelays | array of [CrawlDelay](#crawldelay) | Optional. Per-host delay overrides. Omitted or empty uses the robots.txt crawl-delay when available, otherwise no extra delay | ### CrawlDelay Configures request pacing for one target host Fields: | Name | Type | Description | | --- | --- | --- | | Host | string | Lowercase target host, or host:port for a non-default port, to which this delay applies | | Delay | string | Per-host wait between requests. Use '0' for no extra delay, an integer for fixed seconds, a range such as '1-5' for a random value from 1 inclusive to 5 exclusive, or 'robots' to use the robots.txt crawl-delay when available | ### CrossDomainAccessConfig Controls whether the crawler follows links whose host differs from the current page host Fields: | Name | Type | Description | | --- | --- | --- | | Policy | [CrossDomainAccessPolicies](#crossdomainaccesspolicies) | Optional. Selects which discovered links may be followed. Set this explicitly when providing CrossDomainAccessConfig; the enum default is None. Omitting the entire CrossDomainAccess config uses Subdomains | ### CrossDomainAccessPolicies Domain scoping modes that determine which discovered links are in bounds Enumeration values: | Name | Description | | --- | --- | | None | Follow only links on the same host as the current page | | Subdomains | Follow links on the same host and its subdomains | | CrossDomains | Follow links to any host | ### RetrievalConfig Configures optional enrollment of downloaded page content into the retrieval index Fields: | Name | Type | Description | | --- | --- | --- | | EnrollInIndex | bool | Optional. When true, convert successfully downloaded page content to markdown chunks and enroll it for retrieval. False is the default. Cached pages are skipped unless Force is true | | Force | bool | Optional. When true, enroll cached page content instead of skipping it. Has no effect when EnrollInIndex is false | | MaxTokensPerChunk | int | Optional. Maximum tokens per indexed chunk. Omitted or null uses 512 | | ContentScopes | array of [RetrievalContentScope](#retrievalcontentscope) | Optional. Ordered URL-path patterns and selectors for enrolled content. The first matching scope supplies the selector; if none matches, the whole page is enrolled | | WaitForEnrolled | bool | Optional. When true, wait up to one minute per page for its snippets to become queryable before continuing. False is the default and returns after submitting enrollment | ### RetrievalContentScope Defines which part of matching pages is enrolled by pairing a URL path glob with a content selector Fields: | Name | Type | Description | | --- | --- | --- | | PathPattern | string | Case-sensitive URL path glob. Use '*' for one path segment or file-name part and '**' for nested folders. Example: /products/**/details/*.html | | Selector | string | Selector for the page content to convert to markdown and enroll when PathPattern matches | Path pattern examples: | URL | Pattern | Matches | | --- | --- | --- | | `https://example.com/path/to/resource` | `*` | Yes | | `https://example.com/path/to/resource` | `/*` | Yes | | `https://example.com/path/to/resource` | `/path/to/resource` | Yes | | `https://example.com/path/to/resource` | `/path/to/*` | Yes | | `https://example.com/path/to/resource` | `/path/*/resource` | Yes | | `https://example.com/path/to/resource` | `/**/res*` | Yes | | `https://example.com/path/to/resource` | `/res*` | No | | `https://example.com/path/to/resource` | `/path/to/RESOURCE` | No | ### Selector Format The selector argument is a selector of the following format: ```CSS|XPATH: selector```. The first part defines the selector type, the second one should be a selector in the corresponding type. Supported types: - [CSS](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_selectors) - [XPATH](https://developer.mozilla.org/en-US/docs/Web/XML/XPath) ### HostConfig Configures host-level refresh behavior used by the downloader Fields: | Name | Type | Description | | --- | --- | --- | | ReinitPeriodSec | int | Optional. Host reinitialization interval in seconds. Reinitialization refreshes robots.txt content and renews cookies. Omitted or null uses 604800 seconds (7 days) | # Processing ## Fetch Downloads one absolute URL with the named job's crawler settings and returns page content when ready, or a task content URL while pending ### Endpoint `POST /api/v3/jobs/{jobName}/fetch` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Request Body Required `application/json` string. Absolute HTTP or HTTPS URL to fetch as a single page under the named job. This does not start link crawling ### Responses #### 200 (Ok) Returns `string` containing the page content. #### 202 (Accepted) Returns no body with a task content URL in `Location`. Retry after the number of seconds in `Retry-After` (currently `1`) until the content is ready. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information #### 404 (Not Found) Job not found #### 422 (Unprocessable Content) There is an issue with processing the page content. Refer to the response text for more information ## Crawl Extracts links from the downloaded page using CrawlParams, applies job scope and maxDepth rules, creates child download tasks, and completes the current task. If processing is pending, poll the crawl-result URL ### Endpoint `POST /api/v3/tasks/{taskId}/crawl` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | taskId | string | **Required.** Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints | ### Request Body Required `application/json` body containing [CrawlParams](#crawlparams). ### Responses #### 200 (OK) Returns `DownloadTask[]` as an array of [DownloadTask](#downloadtask). #### 202 (Accepted) Task processing is pending. Follow the `Location` header to the crawl-result or scrape-result endpoint and retry after the number of seconds in `Retry-After` (currently `1`). Continue until the response is no longer 202 (Accepted). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information. #### 403 (Forbidden) Access restricted. Refer to the response text for more information. #### 404 (Not Found) Task not found. #### 422 (Unprocessable Content) There is an issue with processing the page content. Refer to the response text for more information. ## Crawl Result Polls or resumes a previous Crawl request for this task using its stored CrawlParams until processing completes ### Endpoint `GET /api/v3/tasks/{taskId}/crawl-result` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | taskId | string | **Required.** Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints | ### Responses #### 200 (OK) Returns `DownloadTask[]` as an array of [DownloadTask](#downloadtask). #### 202 (Accepted) Task processing is pending. Follow the `Location` header to the crawl-result or scrape-result endpoint and retry after the number of seconds in `Retry-After` (currently `1`). Continue until the response is no longer 202 (Accepted). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information. #### 403 (Forbidden) Access restricted. Refer to the response text for more information. #### 404 (Not Found) Task not found. #### 422 (Unprocessable Content) There is an issue with processing the page content. Refer to the response text for more information. ## Scrape Extracts named values from the downloaded page using ScrapeParams selectors and optional conversions, then completes the current task. If processing is pending, poll the scrape-result URL ### Endpoint `POST /api/v3/tasks/{taskId}/scrape` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | taskId | string | **Required.** Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints | ### Request Body Required `application/json` body containing an array of [ScrapeParams](#scrapeparams). ### Responses #### 200 (OK) Returns `TaskScrapeResult[]`. Each item uses the public [ScrapeResult](#scraperesult) shape. #### 202 (Accepted) Task processing is pending. Follow the `Location` header to the crawl-result or scrape-result endpoint and retry after the number of seconds in `Retry-After` (currently `1`). Continue until the response is no longer 202 (Accepted). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information. #### 403 (Forbidden) Access restricted. Refer to the response text for more information. #### 404 (Not Found) Task not found. #### 422 (Unprocessable Content) There is an issue with processing the page content. Refer to the response text for more information. ### Selector Format The selector argument is a selector of the following format: ```CSS|XPATH: selector```. The first part defines the selector type, the second one should be a selector in the corresponding type. Supported types: - [CSS](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_selectors) - [XPATH](https://developer.mozilla.org/en-US/docs/Web/XML/XPath) ## Scrape Result Polls or resumes a previous Scrape request for this task using its stored ScrapeParams until processing completes ### Endpoint `GET /api/v3/tasks/{taskId}/scrape-result` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | taskId | string | **Required.** Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints | ### Responses #### 200 (OK) Returns `TaskScrapeResult[]`. Each item uses the public [ScrapeResult](#scraperesult) shape. #### 202 (Accepted) Task processing is pending. Follow the `Location` header to the crawl-result or scrape-result endpoint and retry after the number of seconds in `Retry-After` (currently `1`). Continue until the response is no longer 202 (Accepted). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information. #### 403 (Forbidden) Access restricted. Refer to the response text for more information. #### 404 (Not Found) Task not found. #### 422 (Unprocessable Content) There is an issue with processing the page content. Refer to the response text for more information. ## Get Content Reads all or part of the downloaded page for a task, typically one created by Fetch. If processing is pending, poll the same content URL ### Endpoint `GET /api/v3/tasks/{taskId}/content` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | taskId | string | **Required.** Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints | ### Query Parameters | Name | Type | Description | | --- | --- | --- | | startIndex | int | Optional. Optional. Zero-based start index of the returned content substring. Omit or pass null to start at 0. A negative value or an index at or beyond the content length returns an empty ContentSpan | | length | int | Optional. Optional. Maximum number of content characters to return from startIndex. Omit or pass null to return all remaining content; a value of 0 or less returns an empty ContentSpan | ### Responses #### 200 (OK) Returns `ContentChunk` as a [ContentChunk](#contentchunk). #### 202 (Accepted) Task processing is pending. Retry the URL in the `Location` header after the number of seconds in `Retry-After` (currently `1`) until the response is no longer 202 (Accepted). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information #### 404 (Not Found) Task not found #### 422 (Unprocessable Content) There is an issue with processing the page content. Refer to the response text for more information ## Download Status Inspects a task's download state and HTTP attempt history to troubleshoot pending, failed, or completed downloads ### Endpoint `GET /api/v3/tasks/{taskId}/download-status` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | taskId | string | **Required.** Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints | ### Responses #### 200 (OK) Returns `DownloadTaskStatus` as a [DownloadTaskStatus](#downloadtaskstatus). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information #### 404 (Not Found) Task not found ## Data Contracts ### DownloadTask Represents one page download task created by job start, single-URL fetch, or link crawling Fields: | Name | Type | Description | | --- | --- | --- | | Id | string | Download task ID | | Url | string | Download task URL | ### CrawlParams Defines how links are extracted from a page and which discovered URLs remain in scope Fields: | Name | Type | Description | | --- | --- | --- | | Selector | string | Selector used to find link-bearing elements. Use 'CSS: ', 'XPATH: ', or '*' to select all anchor elements | | AttributeName | string | Optional. Attribute read from each matched element to obtain a link. Use 'val' for inner text. Omitted, null, or empty uses href | | MaxDepth | int | Optional. Maximum URL path depth for discovered links. For example, example.com and example.com/index.html have depth 0, while example.com/path/ has depth 1. Omitted or null applies no depth limit | ### ScrapeParams Defines one named field extracted from pages by selector, attribute, and optional conversion Fields: | Name | Type | Description | | --- | --- | --- | | Name | string | Name of the output field populated from this extraction rule. Names must be unique within one scrape request or Traversal level | | Selector | string | Selector used to read matching elements. Use 'CSS: ' or 'XPATH: '. Use '*' to target the whole page; when '*' is used, AttributeName is ignored | | AttributeName | string | Optional. Attribute read from each matched element. Use 'val', or omit/pass null or empty, to return inner text | | Convert | string | Optional. Conversion applied to scraped values. Use 'md()' for Markdown or 'sr()' for main readable content using Mozilla Readability. Omitted or null applies no conversion | ### ScrapeResult Values extracted for one ScrapeParams entry Fields: | Name | Type | Description | | --- | --- | --- | | Name | string | Optional. Field name copied from the matching ScrapeParams entry | | Values | array of string | Extracted values returned by the selector, attribute, and optional conversion | ### ContentChunk Reports the full downloaded content length and the full or requested content span for one task Fields: | Name | Type | Description | | --- | --- | --- | | ContentLength | int | Optional. Length of the full downloaded page content in characters | | ContentSpan | string | Optional. Full downloaded content or the requested substring. An invalid or non-positive range produces an empty string | ### DownloadTaskStatus Current state and request history for a download task Fields: | Name | Type | Description | | --- | --- | --- | | Url | string | URL associated with the download task | | State | [DownloadTaskState](#downloadtaskstate) | Current download task state | | Result | [DownloadInfo](#downloadinfo) | Optional. Final download attempt details, or null when no attempt result is available | | IntermedResults | array of [DownloadInfo](#downloadinfo) | Optional. Earlier attempt details such as redirects or proxy retries. Null means no attempt history is available; an empty array means only the final result exists | ### DownloadTaskState Lifecycle states for a download task Enumeration values: | Name | Description | | --- | --- | | Handled | The downloader finished handling the task. When DownloadTaskStatus.Result is present, inspect IsSuccess and HttpStatusCode to determine request success | | AccessDeniedForRobots | The target URL was blocked by robots.txt | | AllRequestGatesExhausted | All request gateways, such as proxies or host IP addresses, were exhausted without a successful response | | Created | The task was created but has not started | | InProgress | The task is currently being downloaded | | Deleted | The task was deleted | ### DownloadInfo HTTP request and response details for one download attempt Fields: | Name | Type | Description | | --- | --- | --- | | Method | string | HTTP method | | Url | string | Request URL | | IsSuccess | bool | Whether the request completed successfully | | HttpStatusCode | int | HTTP response status code | | ReasonPhrase | string | HTTP reason phrase | | RequestHeaders | array of [HttpHeader](#httpheader) | HTTP headers sent with the request | | ResponseHeaders | array of [HttpHeader](#httpheader) | HTTP headers received in the response | | RequestCookies | array of [Cookie](#cookie) | Cookies sent with the request | | ResponseCookies | array of [Cookie](#cookie) | Cookies received in the response | | RequestDateUtc | datetime | Date and time when the request was sent, in UTC | | DownloadTimeSec | double | Download time in seconds | | ViaProxy | bool | Whether the request was made through a proxy | | WaitTimeSec | double | Total delay in seconds before the request was executed | | CrawlDelaySec | int | Crawl-delay portion of the wait time, in seconds | ### HttpHeader Represents one HTTP header with its name and one or more values Fields: | Name | Type | Description | | --- | --- | --- | | Name | string | HTTP header name, for example User-Agent or Authorization | | Values | array of string | One or more values for this header | ### Cookie HTTP cookie sent with a request or received in a response See [HTTP cookies](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies) for protocol-level background. Fields: | Name | Type | Description | | --- | --- | --- | | Name | string | Cookie name | | Value | string | Optional. Cookie value, or null when absent | | Domain | string | Optional. Cookie domain attribute, or null when absent | | Path | string | Optional. Cookie path attribute, or null when absent | | HttpOnly | bool | Whether the cookie has the HttpOnly attribute | | Secure | bool | Whether the cookie has the Secure attribute | | Expires | datetime | Optional. Cookie expiration date and time, or null for a session cookie or unknown expiration | # Retrieval ## Tenant-wide Search Searches retrieval-indexed crawled content across all jobs in the current tenant to find evidence relevant to a question Requires the `Retrieval` feature. See [Plans](https://webdatasource.com/pricing.html) for feature availability. ### Endpoint `POST /api/v3/jobs/retrieve` ### Request Body Required `application/json` body containing a [RetrieveRequest](#retrieverequest). ### Similarity Thresholds Choose a preset for quick, predictable relevance, or provide a numeric value. Presets map to cosine similarity scores. | Name | When to use | | ------------------ | ---------------------------------------------------------------------------------------- | | exact-match | The query and result describe essentially the same thing, exact term, or strong synonym. | | ------------------ | ---------------------------------------------------------------------------------------- | | same-category | Not identical, but clearly the same family/category and very relevant. | | ------------------ | ---------------------------------------------------------------------------------------- | | same-domain | Topically aligned within the same thematic domain; balanced recall vs. precision. | | ------------------ | ---------------------------------------------------------------------------------------- | | generic-similarity | Broad lexical similarity; maximize recall when you will filter results later. | | ------------------ | ---------------------------------------------------------------------------------------- | ### Responses #### 200 (OK) Returns `RetrievalItem[]` as an array of [RetrievalItem](#retrievalitem). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information. #### 403 (Forbidden) Access restricted. Refer to the response text for more information #### 500 (Internal Server Error) The retrieval operation failed in a backing service. Refer to the response text for more information. ## Job-scoped Search Searches retrieval-indexed crawled content produced by one named job to find evidence relevant to a question Requires the `Retrieval` feature. ### Endpoint `POST /api/v3/jobs/{jobName}/retrieve` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Request Body Required `application/json` body containing a [RetrieveRequest](#retrieverequest). ### Similarity Thresholds Choose a preset for quick, predictable relevance, or provide a numeric value. Presets map to cosine similarity scores. | Name | When to use | | ------------------ | ---------------------------------------------------------------------------------------- | | exact-match | The query and result describe essentially the same thing, exact term, or strong synonym. | | ------------------ | ---------------------------------------------------------------------------------------- | | same-category | Not identical, but clearly the same family/category and very relevant. | | ------------------ | ---------------------------------------------------------------------------------------- | | same-domain | Topically aligned within the same thematic domain; balanced recall vs. precision. | | ------------------ | ---------------------------------------------------------------------------------------- | | generic-similarity | Broad lexical similarity; maximize recall when you will filter results later. | | ------------------ | ---------------------------------------------------------------------------------------- | ### Responses #### 200 (OK) Returns `RetrievalItem[]` as an array of [RetrievalItem](#retrievalitem). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information. #### 403 (Forbidden) Access restricted. Refer to the response text for more information #### 404 (Not Found) The specified job was not found. #### 500 (Internal Server Error) The retrieval operation failed in a backing service. Refer to the response text for more information. ## Data Contracts ### RetrieveRequest Defines a semantic retrieval query, result limit, and score threshold for indexed crawled content Fields: | Name | Type | Description | | --- | --- | --- | | Query | string | Search query used to find relevant indexed content chunks | | Limit | int | Optional. Maximum number of retrieval results to return. Omitted or null uses 10 | | Threshold | string | Optional. Retrieval score threshold. Use an invariant-culture float from 0 to 1 or one of: exact-match, same-category, same-domain, generic-similarity. Omitted, null, or unrecognized text uses same-domain | ### RetrievalItem One matched indexed content chunk returned by retrieval Fields: | Name | Type | Description | | --- | --- | --- | | Score | float | Relevance score for this retrieval match; higher values are returned first | | Span | string | Matched text span with semantic context | | DownloadTasks | array of [DownloadTaskInfo](#downloadtaskinfo) | Downloaded pages that contain the matched text | ### DownloadTaskInfo Source download task for a retrieval match Fields: | Name | Type | Description | | --- | --- | --- | | DownloadTaskId | string | Download task ID | | Url | string | Source page URL | | CaptureDateUtc | datetime | Date and time when the source page was captured, in UTC | # Traversal ## Upsert Traversal Config Validates and stores the named job's TraversalConfig multi-level crawl and scrape plan in the Traversal runner. This does not start a run ### Endpoint `POST /api/v3/jobs/{jobName}/traversal/config` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Request Body Required `application/json` body containing a [TraversalConfig](#traversalconfig). ### Responses #### 204 (No Content) Returns `void` with no response body. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Get Traversal Config Reads the named job's saved TraversalConfig so its multi-level crawl and scrape plan can be inspected or reused ### Endpoint `GET /api/v3/jobs/{jobName}/traversal/config` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Responses #### 200 (Ok) Returns `TraversalConfig?` as a [TraversalConfig](#traversalconfig), or `null` when no configuration is saved. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Delete Traversal Deletes the named job's saved TraversalConfig, schedule, tasks, and run history from the Traversal runner. This does not delete the job ### Endpoint `DELETE /api/v3/jobs/{jobName}/traversal` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Responses #### 204 (No Content) Returns `void` with no response body. #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Start Traversal Run Requests a Traversal run using the named job's current job config and TraversalConfig, reusing an active run instead of starting a duplicate. Use the returned run number for later inspection ### Endpoint `POST /api/v3/jobs/{jobName}/traversal/start` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Responses #### 200 (Ok) Returns `TraversalRunResult` as a [TraversalRunResult](#traversalrunresult). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## List Traversal Runs Lists recent Traversal run summaries for the named job, newest first, so clients can find run numbers and completion state ### Endpoint `GET /api/v3/jobs/{jobName}/traversal/runs` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Query Parameters | Name | Type | Description | | --- | --- | --- | | limit | int | Optional. Default: `10`. Optional. Number of newest runs to return, ordered by run number descending. Default: 10 | ### Responses #### 200 (Ok) Returns `TraversalInfo[]` as an array of [TraversalInfo](#traversalinfo). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Get Traversal Run Info Inspects the progress and configuration used by one Traversal run ### Endpoint `GET /api/v3/jobs/{jobName}/traversal/runs/{runNum}/info` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | | runNum | unsigned long | **Required.** Traversal run number used to read run status, errors, and data for the same job | ### Responses #### 200 (Ok) Returns `TraversalInfo` as a [TraversalInfo](#traversalinfo). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Get Traversal Run Errors Investigates failed downloads from one Traversal run and the parent pages that discovered them ### Endpoint `GET /api/v3/jobs/{jobName}/traversal/runs/{runNum}/errors` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | | runNum | unsigned long | **Required.** Traversal run number used to read run status, errors, and data for the same job | ### Responses #### 200 (Ok) Returns `TraversalErrors` as [TraversalErrors](#traversalerrors). #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Get Traversal Run Data Reads a batch of scraped data produced by a Traversal run. Continue with the returned cursor until no cursor remains ### Endpoint `GET /api/v3/jobs/{jobName}/traversal/runs/{runNum}/data` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | | runNum | unsigned long | **Required.** Traversal run number used to read run status, errors, and data for the same job | ### Query Parameters | Name | Type | Description | | --- | --- | --- | | path | string | Optional. Optional. Path to a TraversalConfig level in the Traversal tree. Use '/' for the root level. Nested paths must start and end with '/', for example '/products/'. Omit to read full leaf-level data for the run. Set a level path to stop at that Traversal level and limit the shape or size of returned JSON objects. See TraversalConfig for level names | | cursor | string | Optional. Optional. Opaque DataCursor returned by the previous response. Omit or pass null to read the first batch | | limit | int | Optional. Default: `10`. Optional. Number of Traversal tasks at the selected path to read for this batch. Default: 10. One task can produce multiple JSON documents, so Data length can be greater than this limit | ### Responses #### 200 (Ok) Returns `TraversalData` as [TraversalData](#traversaldata). Continue with `DataCursor` until the returned cursor is `null`. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Data Contracts ### TraversalConfig Multi-level crawl and scrape plan that defines level names, link selectors, field extraction rules, and nested branches Fields: | Name | Type | Description | | --- | --- | --- | | Name | string | Name of this Traversal level. Use '/' for the root; use names ending in '/', such as 'products/', for branch levels so their data paths are addressable | | CrawlParams | array of [CrawlParams](#crawlparams) | Optional. Link extraction rules used to discover more pages at this level. Omitted, null, or empty performs no same-level link discovery | | ScrapeParams | array of [ScrapeParams](#scrapeparams) | Optional. Field extraction rules used to collect data at this level. Omitted, null, or empty extracts no fields; names must be unique within the level | | Branches | array of [TraversalBranch](#traversalbranch) | Optional. Nested Traversal branches reachable from this level. Omitted, null, or empty defines no child levels | ### CrawlParams Defines how links are extracted from a page and which discovered URLs remain in scope Fields: | Name | Type | Description | | --- | --- | --- | | Selector | string | Selector used to find link-bearing elements. Use 'CSS: ', 'XPATH: ', or '*' to select all anchor elements | | AttributeName | string | Optional. Attribute read from each matched element to obtain a link. Use 'val' for inner text. Omitted, null, or empty uses href | | MaxDepth | int | Optional. Maximum URL path depth for discovered links. For example, example.com and example.com/index.html have depth 0, while example.com/path/ has depth 1. Omitted or null applies no depth limit | ### ScrapeParams Defines one named field extracted from pages by selector, attribute, and optional conversion Fields: | Name | Type | Description | | --- | --- | --- | | Name | string | Name of the output field populated from this extraction rule. Names must be unique within one scrape request or Traversal level | | Selector | string | Selector used to read matching elements. Use 'CSS: ' or 'XPATH: '. Use '*' to target the whole page; when '*' is used, AttributeName is ignored | | AttributeName | string | Optional. Attribute read from each matched element. Use 'val', or omit/pass null or empty, to return inner text | | Convert | string | Optional. Conversion applied to scraped values. Use 'md()' for Markdown or 'sr()' for main readable content using Mozilla Readability. Omitted or null applies no conversion | ### TraversalBranch Describes a child Traversal branch and the link extraction rule used to reach it from the current level `TraversalBranch` inherits the fields of [TraversalConfig](#traversalconfig) and adds this field: | Name | Type | Description | | --- | --- | --- | | EntryRule | [CrawlParams](#crawlparams) | Link extraction rule that discovers pages belonging to this child branch | ### TraversalRunResult Result returned when starting or attaching to a Traversal run Fields: | Name | Type | Description | | --- | --- | --- | | RunNum | long | Traversal run number | | IsNew | bool | True when a new run was started; false when an existing in-progress run was returned | ### TraversalInfo Status, counters, and config snapshot for one Traversal run Fields: | Name | Type | Description | | --- | --- | --- | | RunNum | long | Traversal run number | | StartDateUtc | datetime | Date and time when the run started, in UTC | | CompleteDateUtc | datetime | Optional. Date and time when the run completed, in UTC; null means the run is still in progress | | InProcessCrawlTasksCount | long | Number of download tasks currently in progress | | SuccessfulDownloadTaskCount | long | Number of download tasks that completed successfully | | FailedDownloadTaskCount | long | Number of failed download tasks | | Config | [TraversalConfig](#traversalconfig) | TraversalConfig snapshot used for this run | ### TraversalErrors Failed download tasks from a Traversal run, grouped by parent page Fields: | Name | Type | Description | | --- | --- | --- | | FailedDownloadTasks | array of [FailedDownloadTask](#faileddownloadtask) | Parent pages and the child download tasks that failed while crawling them | ### FailedDownloadTask Failed child download tasks discovered from one parent page Fields: | Name | Type | Description | | --- | --- | --- | | ParentDownloadTaskUrl | string | URL of the parent page where the failed links were discovered, or the literal 'root' for failed start tasks | | FailedDownloadTasks | array of [DownloadTask](#downloadtask) | Child download tasks that failed | ### DownloadTask Represents one page download task created by job start, single-URL fetch, or link crawling Fields: | Name | Type | Description | | --- | --- | --- | | Id | string | Download task ID | | Url | string | Download task URL | ### TraversalData Paged scraped data returned for a Traversal run Fields: | Name | Type | Description | | --- | --- | --- | | Data | array of string | Scraped data records serialized as JSON strings | | DataCursor | string | Optional. Opaque cursor for fetching the next batch of data; null means there is no next page | # Traversal Scheduling ## Upsert Traversal Schedule Creates or replaces the named job's TraversalSchedule for automatic runs. A saved TraversalConfig is required. This configures future runs and does not start one immediately Requires the `Traversal Scheduling` feature. Scheduling is available starting with the [Business plan](https://webdatasource.com/pricing.html). ### Endpoint `POST /api/v3/jobs/{jobName}/traversal/schedule` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Request Body Required `application/json` body containing a [TraversalSchedule](#traversalschedule). ### Responses #### 204 (No Content) Returns `void` with no response body. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Get Traversal Schedule Reads the named job's saved automatic Traversal schedule. A saved TraversalConfig is required Requires the `Traversal Scheduling` feature. Scheduling is available starting with the Business plan. ### Endpoint `GET /api/v3/jobs/{jobName}/traversal/schedule` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Responses #### 200 (Ok) Returns `TraversalSchedule?` as a [TraversalSchedule](#traversalschedule), or `null` when no schedule is saved. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Delete Traversal Schedule Deletes the named job's TraversalSchedule and stops future scheduled starts. A saved TraversalConfig is required. This does not delete the job, Traversal config, or previous runs Requires the `Traversal Scheduling` feature. Scheduling is available starting with the Business plan. ### Endpoint `DELETE /api/v3/jobs/{jobName}/traversal/schedule` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | jobName | string | **Required.** Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data | ### Responses #### 204 (No Content) Returns `void` with no response body. #### 400 (Bad Request) Invalid request parameters. Refer to the response text for more information #### 403 (Forbidden) Access restricted. Refer to the response text for more information ## Data Contracts ### TraversalSchedule Configures automatic Traversal runs for a job Fields: | Name | Type | Description | | --- | --- | --- | | CronExpression | string | Five-field cron expression evaluated in UTC after the previous run completes: minute, hour, day of month, month, day of week. For example, '0 0 * * *' schedules midnight UTC | | Enabled | bool | Optional. When true, allow automatic runs according to CronExpression. False disables scheduled starts without preventing manual runs | # Tenants ## Delete Tenant Deletes a tenant and cleans up its jobs, download tasks, Traversal configs, schedules, tasks and run history, cached and scraped data, and retrieval-index data across backing services ### Endpoint `DELETE /api/v3/tenants/{tenantId}` ### Path Parameters | Name | Type | Description | | --- | --- | --- | | tenantId | string | **Required.** Tenant identifier whose data should be deleted | ### Responses #### 204 (No Content) Returns `void` with no response body after deleting the tenant. ## Data Contracts This endpoint has no request or response data contracts. # Deployment Methods # Overview WDS can be deployed as one Solidstack container or as a distributed service topology. Docs and Playground are optional auxiliary containers; they are not part of the eight-service WDS server inventory. ## Options at a glance - **Docker** runs only Solidstack and uses external MongoDB. It is the smallest local evaluation option. - **Docker Compose** provides reconciled MINI and OEM Business topologies. BOX Free and BOX Business remain documented but require current bundled-infrastructure support files and release image values before their examples are runnable. Compose options also include Docs and Playground. - **Helm** supports single-service and multi-service Kubernetes deployments with independent scaling and health management. - **Air-gapped** deployment mirrors the required images to a private registry before using Helm or Compose without Internet access. ## Prerequisites - Docker deployments need Docker and a reachable MongoDB database. - Compose deployments need Docker Compose. MINI requires reachable external MongoDB and embedding services. OEM Business additionally requires external database, search, and cache services. The BOX variants are pending bundled-infrastructure reconciliation. - Helm deployments need a Kubernetes cluster, Helm or Terraform, and the configured database and optional search/cache/embedding services. - Air-gapped deployments need a reachable private registry and permission to mirror every selected image. ## Detailed guides - [](/releases/latest/server/deployments/helm.txt) - [Docker](/releases/latest/server/deployments/docker.html) - [Docker Compose](/releases/latest/server/deployments/dockercompose.html) - [Helm Chart](/releases/latest/server/deployments/helm.html) - [Air-Gapped](/releases/latest/server/deployments/airgapped.html) # WDS API Server The WDS API Server provides the public REST and MCP surfaces for configuring jobs, downloading and processing web resources, running Traversals, and retrieving indexed content. ## Capabilities - Store job configurations, start jobs, fetch pages, and inspect job state. - Crawl links, scrape structured values, read downloaded content in chunks, and inspect task status. - Define, run, inspect, and schedule Traversals. - Index and retrieve content with full-text or vector search. - Expose the same workflows to AI clients through the [MCP server](../mcp/index.html). ## Services The current server consists of eight `.Web` projects: Dapi, Crawler, Datakeeper, Scraper, Idealer, Retriever, Jober, and Solidstack. Solidstack runs the gateway and backend capabilities in one process; the other projects form the independently deployable multi-service topology. See [Services](./services/index.html) for responsibilities, dependencies, ports, and configuration. ## Deployment options - [Docker](./deployments/docker.html) runs Solidstack as a single container for evaluation and small workloads. - [Docker Compose](./deployments/dockercompose.html) provides current MINI and OEM Business topologies; the bundled-infrastructure BOX examples are pending their required support files and release image values. - [Helm](./deployments/helm.html) deploys either the single-service or multi-service topology to Kubernetes. - [Air-gapped deployment](./deployments/airgapped.html) covers private registries and offline environments. ## Entry points - API: `/api/v3`; see [API](./api/index.html). - Swagger UI and OpenAPI: `/api/swagger` and `/api/swagger/swagger.json`. - MCP Streamable HTTP endpoint: `/mcp`; see [MCP](../mcp/index.html). - Health probes: `/health` and `/ready`. - Auxiliary Playground and local Docs endpoints depend on the selected deployment topology. ## Next steps - Choose a [deployment method](./deployments/index.html). - Follow the API [Quickstart](./api/quickstart.html). - Review the [service architecture](./services/index.html). # Services # Overview The WDS server is built from eight `.Web` service projects. It can run as one in-process service or as a distributed service set. ## Service inventory | Service | Responsibility | | --- | --- | | [Dapi](./dapi.html) | Public REST, Swagger, and MCP gateway; job persistence and orchestration. | | [Crawler](./crawler.html) | HTTP downloads, request controls, and Crawler address registration. | | [Datakeeper](./datakeeper.html) | Job settings, download-task state, downloaded content, and cache storage. | | [Scraper](./scraper.html) | Structured extraction, content conversion, and scraped-data storage. | | [Idealer](./idealer.html) | Stable identifier allocation and tenant cleanup. | | [Retriever](./retriever.html) | Full-text and vector indexing and search. | | [Jober](./jober.html) | Traversal configuration, execution, scheduling, recovery, and results. | | [Solidstack](./solidstack.html) | Single-container implementation of the public gateway and backend capabilities. | Docs and Playground are auxiliary deployment containers, not WDS server services. Compose and Helm deployments may include them for offline documentation and repeatable examples; see [Auxiliary containers](../deployments/dockercompose.html#auxiliary-containers). ## Single-service mode [Solidstack](./solidstack.html) runs the Dapi, Crawler, Datakeeper, Idealer, Jober, Scraper, and Retriever implementations in one process. It is suited to evaluation and small workloads, but its components cannot be scaled or restarted independently. The current Solidstack feature provider does not enable Scheduling. See [Plans](https://webdatasource.com/pricing.html) for feature availability. ## Multi-service mode Multi-service deployments run [Dapi](./dapi.html), [Crawler](./crawler.html), [Datakeeper](./datakeeper.html), [Scraper](./scraper.html), [Idealer](./idealer.html), [Retriever](./retriever.html), and [Jober](./jober.html) as separate processes. This topology supports independent scaling, health monitoring, resource allocation, and failure isolation. It is available with the [Business plan](https://webdatasource.com/pricing.html). Dapi is the public entry point. It stores job configurations in MongoDB and calls the other services over gRPC-Web. Crawler registers with Datakeeper Resource Manager; Datakeeper, Scraper, and Retriever use Idealer; and Jober coordinates Dapi, Datakeeper, and Scraper for Traversal runs. ## Runtime dependencies - MongoDB stores service state. Every distributed service except Crawler uses it; Solidstack uses one MongoDB configuration for its in-process components. - Datakeeper can use a separate MongoDB or S3-compatible cache. Without one, it uses its primary MongoDB database. - Retriever can use a separate MongoDB Atlas Search database. Vector modes also require an HTTP embedding service. - Dapi, Crawler, Datakeeper, Idealer, Jober, Scraper, and Retriever require service-specific license configuration in multi-service mode. # Dapi Service Dapi is the public gateway for WDS. It exposes the API, Swagger UI, and MCP server; persists job configurations; routes work to the backend services; propagates feature flags; and coordinates tenant cleanup. Release image: [`webdatasource/dapi:v3.0.0`](https://hub.docker.com/repository/docker/webdatasource/dapi/general) ## Configuration | Name | Required | Default | Meaning | | --- | --- | --- | --- | | `MONGODB_CONNECTION_STRING` | Yes | None | [MongoDB connection string](https://www.mongodb.com/docs/manual/reference/connection-string/) for Dapi state. Include a database name unless `MONGODB_DATABASE_NAME` is set. | | `MONGODB_DATABASE_NAME` | No | Database from the connection string | Overrides the MongoDB database name. | | `DATAKEEPER_ORIGIN` | Yes | None | Origin of [Datakeeper](./datakeeper.html); Dapi appends port `8082` for gRPC-Web. | | `SCRAPER_ORIGIN` | Yes | None | Origin of [Scraper](./scraper.html); Dapi appends port `8082` for gRPC-Web. | | `IDEALER_ORIGIN` | Yes | None | Origin of [Idealer](./idealer.html); Dapi appends port `8082` for gRPC-Web. | | `JOBER_ORIGIN` | Yes | None | Origin of [Jober](./jober.html); Dapi appends port `8082` for gRPC-Web. | | `RETRIEVER_ORIGIN` | Yes | None | Origin of [Retriever](./retriever.html); Dapi appends port `8082` for gRPC-Web. | | `JOB_TYPES` | Yes | None | Available [job types](#jobtypes), supplied as a comma-separated value. | | `TASKS_GET_RESULT_RETRY_DELAY_MS` | No | `1000` | Delay in milliseconds between attempts to obtain a downstream task result. | | `FEATURE_FLAG_RETRIEVAL_ENABLED` | No | `false` | Enables the Retrieval surface before license checks are applied. | | `FEATURE_FLAG_SCHEDULING_ENABLED` | No | `false` | Enables the Scheduling surface before license checks are applied. | | `LICENSE_KEY` | Yes | None | WDS license key. See [Plans](https://webdatasource.com/pricing.html). | | `BASE_PATH` | No | Root path | Path base when Dapi is hosted under a reverse-proxy subpath. | | `GLOBAL_EXCEPTION_RESPONSE_DELAY_MS` | No | `1000` | Delay in milliseconds before returning an unhandled-error response. | | `HEALTH_PROBE_LOG_LEVEL` | No | `Debug` | Log level used for `/health` and `/ready` request messages. | | `MIN_LOG_LEVEL` | No | `Info` | Minimum application log level. | ## Endpoints and dependencies Port `8080` serves the API, `/api/swagger`, `/api/swagger/swagger.json`, `/mcp`, `/health`, and `/ready`. Port `8082` serves Dapi gRPC-Web over HTTP/1.1. Dapi uses MongoDB and connects to Datakeeper, Scraper, Idealer, Jober, and Retriever over gRPC-Web. ### JobTypes Supported job execution environments. Dapi or Solidstack restricts the allowed values through `JOB_TYPES`; Crawler or Solidstack must also advertise a matching address configuration. | Name | Description | | -------- | ------------------------------------------------------------------------------------------------ | | internet | Crawl data from internet sources via request gateways (Proxy addresses, Host IP addresses, etc.) | | intranet | Crawl data from intranet sources with no limits | # Crawler Service Crawler downloads pages for running jobs. It executes HTTP requests, applies proxy, cookie, HTTPS, delay, and throttling settings, and registers its usable addresses with Datakeeper Resource Manager. Release image: [`webdatasource/crawler:v3.0.0`](https://hub.docker.com/repository/docker/webdatasource/crawler/general) ## Configuration | Name | Required | Default | Meaning | | --- | --- | --- | --- | | `DATAKEEPER_ORIGIN` | Yes | None | Origin of [Datakeeper](./datakeeper.html); Crawler appends port `8083` for the Resource Manager gRPC-Web endpoint. | | `GRPC_DNS_RESOLVER_REFRESH_INTERVAL_SEC` | No | `15` | DNS resolver refresh interval in seconds for the Datakeeper gRPC-Web connection. | | `SERVICE_HOST` | Yes | None | Host advertised for this Crawler. It is combined with `http://` and port `8082` to form the registered gRPC-Web URL. | | `EXTERNAL_IP_ADDRESS_CONFIGS` | No | None | Comma-separated [external-address configurations](#external-ip-getter-services). Supply at least one usable entry when this Crawler should accept intranet or Internet work. | | `MAX_INACTIVE_SEC_TO_REREGISTRAR` | No | `60` | Inactive period in seconds before Crawler registers itself again with Datakeeper Resource Manager. | | `LICENSE_KEY` | Yes | None | WDS license key. See [Plans](https://webdatasource.com/pricing.html). | | `HEALTH_PROBE_LOG_LEVEL` | No | `Debug` | Log level used for `/health` and `/ready` request messages. | | `MIN_LOG_LEVEL` | No | `Info` | Minimum application log level. | ## Endpoints and dependencies Port `8080` serves `/health` and `/ready`. Port `8082` serves Crawler gRPC-Web over HTTP/1.1. Crawler connects to Datakeeper Resource Manager on port `8083` and periodically registers its own port-`8082` address. ## External IP getter services For Internet jobs, requests can be sent through proxies or from registered Crawler addresses. The supported configuration values are: - `amazon`: resolve the public address through `https://checkip.amazonaws.com`. - `intranet`: register the Crawler for intranet jobs. - A literal IP address, such as `20.21.22.23`: register that address directly. ### Examples Intranet jobs only: ```bash EXTERNAL_IP_ADDRESS_CONFIGS=intranet ``` Intranet and Internet jobs with automatic public-address discovery: ```bash EXTERNAL_IP_ADDRESS_CONFIGS=intranet,amazon ``` Internet jobs with a fixed public address: ```bash EXTERNAL_IP_ADDRESS_CONFIGS=20.21.22.23 ``` # Datakeeper Service Datakeeper stores job settings and downloaded content, manages Crawler registrations and download tasks, and reuses cached pages when their HTTP validators permit it. Release image: [`webdatasource/datakeeper:v3.0.0`](https://hub.docker.com/repository/docker/webdatasource/datakeeper/general) ## Configuration | Name | Required | Default | Meaning | | --- | --- | --- | --- | | `MONGODB_CONNECTION_STRING` | Yes | None | [MongoDB connection string](https://www.mongodb.com/docs/manual/reference/connection-string/) for Datakeeper state. Include a database name unless `MONGODB_DATABASE_NAME` is set. | | `MONGODB_DATABASE_NAME` | No | Database from the connection string | Overrides the MongoDB database name. | | `CACHE_CONNECTION_STRING` | No | Primary MongoDB connection | Selects a separate MongoDB or S3-compatible content cache. See [Caching](#caching). | | `CACHE_DATABASE_NAME` | No | Database from the cache connection string | Overrides the cache database name for database-backed caches. | | `IDEALER_ORIGIN` | Yes | None | Origin of [Idealer](./idealer.html); Datakeeper appends port `8082` for gRPC-Web. | | `LICENSE_KEY` | Yes | None | WDS license key. See [Plans](https://webdatasource.com/pricing.html). | | `HEALTH_PROBE_LOG_LEVEL` | No | `Debug` | Log level used for `/health` and `/ready` request messages. | | `MIN_LOG_LEVEL` | No | `Info` | Minimum application log level. | ## Endpoints and dependencies Port `8080` serves `/health` and `/ready`. Port `8082` serves the main Datakeeper gRPC-Web API over HTTP/1.1, and port `8083` serves Datakeeper Resource Manager gRPC-Web over HTTP/1.1. Datakeeper uses MongoDB, connects to Idealer, and optionally connects to a separate cache provider. ## Caching Downloaded pages are cached so later scraping can reuse their content without repeating unnecessary requests. When a resource supplies `ETag` or `Last-Modified`, Datakeeper preserves the validators so Crawler can make conditional requests and reuse still-current content. Without `CACHE_CONNECTION_STRING`, Datakeeper stores cached pages in its primary MongoDB database. The supported external cache forms are: - MongoDB: `mongodb://:@:27017/` or `mongodb+srv://:@.mongodb.net/`. - S3-compatible storage: `s3://:@:/?ssl=true|false`. # Scraper Service Scraper extracts structured values from downloaded HTML, converts content into supported formats, and stores and retrieves scraped data by identifier. Release image: [`webdatasource/scraper:v3.0.0`](https://hub.docker.com/repository/docker/webdatasource/scraper/general) ## Configuration | Name | Required | Default | Meaning | | --- | --- | --- | --- | | `MONGODB_CONNECTION_STRING` | Yes | None | [MongoDB connection string](https://www.mongodb.com/docs/manual/reference/connection-string/) for scraped data. Include a database name unless `MONGODB_DATABASE_NAME` is set. | | `MONGODB_DATABASE_NAME` | No | Database from the connection string | Overrides the MongoDB database name. | | `IDEALER_ORIGIN` | Yes | None | Origin of [Idealer](./idealer.html); Scraper appends port `8082` for gRPC-Web. | | `LICENSE_KEY` | Yes | None | WDS license key. See [Plans](https://webdatasource.com/pricing.html). | | `HEALTH_PROBE_LOG_LEVEL` | No | `Debug` | Log level used for `/health` and `/ready` request messages. | | `MIN_LOG_LEVEL` | No | `Info` | Minimum application log level. | ## Endpoints and dependencies Port `8080` serves `/health` and `/ready`. Port `8082` serves Scraper gRPC-Web over HTTP/1.1. Scraper uses MongoDB and connects to Idealer over gRPC-Web; Dapi and Jober send it extraction work. # Idealer Service Idealer allocates and persists stable identifiers used by the other WDS services and participates in tenant cleanup. Release image: [`webdatasource/idealer:v3.0.0`](https://hub.docker.com/repository/docker/webdatasource/idealer/general) ## Configuration | Name | Required | Default | Meaning | | --- | --- | --- | --- | | `MONGODB_CONNECTION_STRING` | Yes | None | [MongoDB connection string](https://www.mongodb.com/docs/manual/reference/connection-string/) for identifier state. Include a database name unless `MONGODB_DATABASE_NAME` is set. | | `MONGODB_DATABASE_NAME` | No | Database from the connection string | Overrides the MongoDB database name. | | `LICENSE_KEY` | Yes | None | WDS license key. See [Plans](https://webdatasource.com/pricing.html). | | `HEALTH_PROBE_LOG_LEVEL` | No | `Debug` | Log level used for `/health` and `/ready` request messages. | | `MIN_LOG_LEVEL` | No | `Info` | Minimum application log level. | ## Endpoints and dependencies Port `8080` serves `/health` and `/ready`. Port `8082` serves Idealer gRPC-Web over HTTP/1.1. Idealer's runtime dependency is MongoDB; Datakeeper, Scraper, and Retriever call it over gRPC-Web. # Retriever Service Retriever provides indexing and search for [job-scoped retrieval](../api/retrieval.html#job-scoped-search) and [cross-job retrieval](../api/retrieval.html#tenant-wide-search). It enrolls crawl results, runs full-text and vector searches, obtains embeddings from a configurable HTTP service, and tracks each job's enrollment state. Release image: [`webdatasource/retriever:v3.0.0`](https://hub.docker.com/repository/docker/webdatasource/retriever/general) ## Configuration | Name | Required | Default | Meaning | | --- | --- | --- | --- | | `MONGODB_CONNECTION_STRING` | Yes | None | [MongoDB connection string](https://www.mongodb.com/docs/manual/reference/connection-string/) for Retriever state. Include a database name unless `MONGODB_DATABASE_NAME` is set. | | `MONGODB_DATABASE_NAME` | No | Database from the connection string | Overrides the primary MongoDB database name. | | `IDEALER_ORIGIN` | Yes | None | Origin of [Idealer](./idealer.html); Retriever appends port `8082` for gRPC-Web. | | `SEARCH_MODE` | No | `FullText` | Search behavior: `FullText`, `Vector`, or `FullTextAndVector`. | | `SEARCH_DB_CONNECTION_STRING` | No | Primary MongoDB connection | Selects a dedicated MongoDB Atlas Search database. | | `SEARCH_DB_DATABASE_NAME` | No | Database from the search connection string | Overrides the dedicated search database name. | | `EMBEDDING_SERVICE_URL` | For vector modes | None | HTTP endpoint used to create embeddings. | | `EMBEDDING_SERVICE_API_KEY` | No | None | Bearer token sent to the embedding service. | | `EMBEDDING_SERVICE_REQUEST_TEMPLATE` | No | `{'model':'embeddinggemma','input':null}` | JSON request template sent to the embedding service. | | `EMBEDDING_SERVICE_CONTENT_JSON_PATH` | No | `$.input` | JSONPath whose value is replaced with the input array. | | `EMBEDDING_SERVICE_RESULT_JSON_PATH` | No | `$.embeddings` | JSONPath used to extract the returned embeddings array. | | `EMBEDDING_VECTORS_LENGTH` | No | `768` | Number of dimensions stored in the MongoDB Atlas vector index. | | `LICENSE_KEY` | Yes | None | WDS license key with the Retrieval feature. See [Plans](https://webdatasource.com/pricing.html). | | `HEALTH_PROBE_LOG_LEVEL` | No | `Debug` | Log level used for `/health` and `/ready` request messages. | | `MIN_LOG_LEVEL` | No | `Info` | Minimum application log level. | ## Endpoints and dependencies Port `8080` serves `/health` and `/ready`. Port `8082` serves Retriever gRPC-Web over HTTP/1.1. Retriever uses its primary MongoDB database, connects to Idealer, and optionally connects to a separate MongoDB Atlas Search database and an HTTP embedding service. ## Search database Retriever currently uses MongoDB Atlas Search for full-text and vector indexes. If `SEARCH_DB_CONNECTION_STRING` is omitted, the primary MongoDB deployment must support the selected Atlas Search mode. Vector modes also require a reachable embedding service and matching `EMBEDDING_VECTORS_LENGTH`. # Jober Service Jober stores Traversal configurations, starts scheduled or on-demand Traversal runs, coordinates crawl and scrape tasks, recovers abandoned crawl work, and returns Traversal results and data cursors. Release image: [`webdatasource/jober:v3.0.0`](https://hub.docker.com/repository/docker/webdatasource/jober/general) ## Configuration | Name | Required | Default | Meaning | | --- | --- | --- | --- | | `MONGODB_CONNECTION_STRING` | Yes | None | [MongoDB connection string](https://www.mongodb.com/docs/manual/reference/connection-string/) for Traversal configuration and run state. Include a database name unless `MONGODB_DATABASE_NAME` is set. | | `MONGODB_DATABASE_NAME` | No | Database from the connection string | Overrides the MongoDB database name. | | `DAPI_ORIGIN` | Yes | None | Origin of [Dapi](./dapi.html); Jober appends port `8082` for gRPC-Web callbacks. | | `DATAKEEPER_ORIGIN` | Yes | None | Origin of [Datakeeper](./datakeeper.html); Jober appends port `8082` for gRPC-Web. | | `SCRAPER_ORIGIN` | Yes | None | Origin of [Scraper](./scraper.html); Jober appends port `8082` for gRPC-Web. | | `TASKS_GET_RESULT_RETRY_DELAY_MS` | No | `1000` | Delay in milliseconds between task-processing iterations and result checks. | | `CRAWL_TASK_FAILURE_RECOVERY_DELAY_SEC` | No | `600` | Age in seconds after which an abandoned captured crawl task can be recovered. | | `LICENSE_KEY` | Yes | None | WDS license key. See [Plans](https://webdatasource.com/pricing.html). | | `HEALTH_PROBE_LOG_LEVEL` | No | `Debug` | Log level used for `/health` and `/ready` request messages. | | `MIN_LOG_LEVEL` | No | `Info` | Minimum application log level. | ## Endpoints and dependencies Port `8080` serves `/health` and `/ready`. Port `8082` serves Jober gRPC-Web over HTTP/1.1. Jober uses MongoDB and connects to Dapi, Datakeeper, and Scraper over gRPC-Web. # Solidstack Service Solidstack is the single-container WDS server. It exposes the Dapi API, Swagger, and MCP surfaces while running Crawler, Datakeeper, Idealer, Jober, Scraper, and Retriever implementations in process. This lowers the footprint for evaluation and small workloads but does not provide the independent scaling or fault isolation of a multi-service deployment. Release image: [`webdatasource/solidstack:v3.0.0`](https://hub.docker.com/repository/docker/webdatasource/solidstack/general) ## Configuration | Name | Required | Default | Meaning | | --- | --- | --- | --- | | `MONGODB_CONNECTION_STRING` | Yes | None | [MongoDB connection string](https://www.mongodb.com/docs/manual/reference/connection-string/) for all in-process components. Include a database name unless `MONGODB_DATABASE_NAME` is set. | | `MONGODB_DATABASE_NAME` | No | Database from the connection string | Overrides the MongoDB database name. | | `JOB_TYPES` | Yes | None | Available [job types](#jobtypes), supplied as a comma-separated value. | | `TASKS_GET_RESULT_RETRY_DELAY_MS` | No | `1000` | Delay in milliseconds between attempts to obtain an in-process task result. | | `CRAWL_TASK_FAILURE_RECOVERY_DELAY_SEC` | No | `600` | Age in seconds after which an abandoned captured crawl task can be recovered. | | `EXTERNAL_IP_ADDRESS_CONFIGS` | No | None | Comma-separated [Crawler address configurations](./crawler.html#external-ip-getter-services). | | `FEATURE_FLAG_RETRIEVAL_ENABLED` | No | `false` | Enables the Retrieval surface. | | `SEARCH_MODE` | No | `FullText` | Search behavior: `FullText`, `Vector`, or `FullTextAndVector`. | | `EMBEDDING_SERVICE_URL` | For vector modes | None | HTTP endpoint used to create embeddings. | | `EMBEDDING_SERVICE_API_KEY` | No | None | Bearer token sent to the embedding service. | | `EMBEDDING_SERVICE_REQUEST_TEMPLATE` | No | `{'model':'embeddinggemma','input':null}` | JSON request template sent to the embedding service. | | `EMBEDDING_SERVICE_CONTENT_JSON_PATH` | No | `$.input` | JSONPath whose value is replaced with the input array. | | `EMBEDDING_SERVICE_RESULT_JSON_PATH` | No | `$.embeddings` | JSONPath used to extract the returned embeddings array. | | `EMBEDDING_VECTORS_LENGTH` | No | `768` | Number of dimensions stored in the MongoDB Atlas vector index. | | `GLOBAL_EXCEPTION_RESPONSE_DELAY_MS` | No | `1000` | Delay in milliseconds before returning an unhandled-error response. | | `HEALTH_PROBE_LOG_LEVEL` | No | `Debug` | Log level used for `/health` and `/ready` request messages. | | `MIN_LOG_LEVEL` | No | `Info` | Minimum application log level. | ## Endpoints and dependencies Port `8080` serves the API, `/api/swagger`, `/api/swagger/swagger.json`, `/mcp`, `/health`, and `/ready`. Solidstack uses MongoDB for system, cache, and search data and optionally calls an HTTP embedding service for vector modes; it has no inter-service gRPC-Web dependencies. When Retrieval is enabled, the primary MongoDB deployment must support MongoDB Atlas Search. Vector modes also require an embedding service, and `EMBEDDING_VECTORS_LENGTH` must match the dimensions configured by the MongoDB vector-search index. Solidstack's current feature provider disables Scheduling. Use the multi-service Dapi and Jober topology when Scheduling is required. ### JobTypes Supported job execution environments. Dapi or Solidstack restricts the allowed values through `JOB_TYPES`; Crawler or Solidstack must also advertise a matching address configuration. | Name | Description | | -------- | ------------------------------------------------------------------------------------------------ | | internet | Crawl data from internet sources via request gateways (Proxy addresses, Host IP addresses, etc.) | | intranet | Crawl data from intranet sources with no limits | # MS SQL Server # CLR Functions # CLR functions and data contracts The SQL installation script publishes eight functions and 23 user-defined types (UDTs) in the `wds` schema. Each function page documents its SQL signature and embeds the complete source-backed UDT contract graph used by its arguments and return value, so the callable operation and its data shape stay together. - [ServerStatus](./server-status.html) checks the configured server's MSSQL-compatible API dependencies. - [Start](./start.html) configures and starts a job, then returns its initial download tasks. - [Crawl](./crawl.html) extracts links and creates child download tasks. - [ScrapeFirst](./scrape-first.html) returns the first selected value. - [ScrapeAll](./scrape-all.html) returns every selected value as rows. - [ScrapeMultiple](./scrape-multiple.html) creates a fluent batch-scrape builder. - [TaskStatus](./task-status.html) reads a task's state and request history. - [ToStringsTable](./to-strings-table.html) projects string items into SQL rows. Follow the in-page type links on each function page for field definitions, initialization behavior, validation rules, and public SQL-facing methods. For example, [Start](./start.html#jobconfig) contains `JobConfig` and its nested configuration contracts, while [TaskStatus](./task-status.html#downloadtaskstatus) contains the task-status result graph. ## Context diagram ![Context diagram](/assets/img/clr-functions/context-diagram.png?fp=UxzORxhteMx6eOBl) ## Table of contents - [Install](/releases/latest/mssql/clr-functions/install.html) - [ServerStatus](/releases/latest/mssql/clr-functions/server-status.html) - [Start](/releases/latest/mssql/clr-functions/start.html) - [Crawl](/releases/latest/mssql/clr-functions/crawl.html) - [ScrapeFirst](/releases/latest/mssql/clr-functions/scrape-first.html) - [ScrapeAll](/releases/latest/mssql/clr-functions/scrape-all.html) - [ScrapeMultiple](/releases/latest/mssql/clr-functions/scrape-multiple.html) - [TaskStatus](/releases/latest/mssql/clr-functions/task-status.html) - [ToStringsTable](/releases/latest/mssql/clr-functions/to-strings-table.html) # Installing CLR library to MS SQL Server The following MS SQL Server versions are supported and tested: - MS SQL Server 2022 running on a Windows machine (Linux-based versions that can be run in a docker container are not supported CLR functions) > **NOTE:** This doesn't mean that the other versions are not supported at all. WDS just haven't been tested with them yet, so follow the releases. To install CLR functions into an SQL Server instance, on the Releases page choose the [latest](https://github.com/webdatasource/wds.mssql.clr/releases/tag/latest) or a [specific version](https://github.com/webdatasource/wds.mssql.clr/releases) and download its Artifacts.zip archive. This archive has the following files: 1. **WDS.MsSql.Clr.dll** - .NET Framework 4.8 assembly with the CLR functions 2. **WDS.MsSql.Clr.hash** - a hash of the WDS.MsSql.Clr.dll that is required for adding the assembly into an SQL Server instance 3. **WdsClrFunctions.sql** - SQL script that adds the functions into an SQL Server instance 4. **Install.bat** - Windows script that configures and runs the WdsClrFunctions.sql against a particular instance of SQL Server. This script is idempotent so it can be run multiple times and all components will be reinstalled from scratch. **WdsClrFunctions.sql** and **Install.bat** scripts are available in the [wds.mssql.clr](https://github.com/webdatasource/wds.mssql.clr) repository for evaluation (see the WDS.MsSql.Scripts directory). **WDS.MsSql.Clr.dll**, and **WDS.MsSql.Clr.hash** files are built by GitHub actions automatically. Nonetheless, these two files can be compiled from the source code (the WDS.MsSql.Clr.hash is created by an after-build script in the WDS.MsSql.Clr project file). In order to add CLR functions in an SQL Server instance, the CLR feature should be enabled. To enable it, the WdsClrFunctions.sql script contains the following section: ``` SQL EXEC sp_configure 'show advanced options', 1; RECONFIGURE; EXEC sp_configure 'clr enable', 1; RECONFIGURE; ``` There is always the [latest](https://github.com/webdatasource/wds.mssql.clr/releases/tag/latest) release and the other releases. For evaluation purposes, the latest release is recommended, but it's better to use specific versions in production and perform updates from version to version according to a release process.\ ## Installation Steps 1. Download an Artifacts.zip archive from one of the releases or the latest one 2. Unarchive the Artifacts.zip and get into the unpacked folder 3. Run the Install.bat script and follow instructions 4. After the script completed successfully, the following message will be shown: ```All done. The MS SQL instance on "server_address" is ready to run WDS CLR functions.``` All [CLR functions and their UDT contracts](./index.html) are installed to the `wds` namespace by default, and all examples use this namespace. This can be changed manually in `WdsClrFunctions.sql` if necessary. ### Install.bat configuration The following environment variables are used to configure the Install.bat script: | Environmetn Valiable | Default Value | Description | | -------------------- | --------------- | ---------------------------------- | | SERVER | localhost | SQL Server instance address | #### Install.bat run examples with overrides 1. With a custom SQL Server address ``` BASH cmd /c "SET SERVER=10.11.12.13 && Install.bat" ``` # ServerStatus Checks whether the configured server's MSSQL-compatible API dependencies are ready and returns a status row for the check. ## Syntax ```sql wds.ServerStatus(@serverConfig) ``` ## Arguments | Name | Type | Description | | --- | --- | --- | | @serverConfig | [wds.ServerConfig](#serverconfig) | Supplies the server URI used to call the MSSQL-compatible readiness endpoint. | ## Returns `TABLE (Name NVARCHAR(255), Value NVARCHAR(MAX), Description NVARCHAR(MAX))` — a table of status indicators. The `Ready` row reports whether the readiness request succeeded; `Description` contains the response text or the error. ## Data Contracts ### ServerConfig API server configuration. | Name | Type | Description | | --- | --- | --- | | Uri | Uri | API server URI. | `Parse` accepts `wds://user:password@host:port?https=false`, requires the `wds` scheme, and produces an HTTP URI unless `https=true` is supplied. Validation requires a URI. # Start Starts or restarts the job named in the supplied configuration after applying its crawler settings, then returns its initial download tasks. ## Syntax ```sql wds.Start(@jobConfig) ``` ## Arguments | Name | Type | Description | | --- | --- | --- | | @jobConfig | [wds.JobConfig](#jobconfig) | Supplies the server connection, job name, start URLs, and crawler settings. Its job name selects the MSSQL-compatible job to configure and start. | ## Returns `TABLE (Task wds.DownloadTask)` — [wds.DownloadTask](#downloadtask) values, one for each configured start URL. ## Data Contracts ### JobConfig Configures a WDS job and its request, crawl, retrieval, and host behavior. | Name | Type | Description | | --- | --- | --- | | Server | [ServerConfig](#serverconfig) | WDS API server connection parameters. | | JobType | string | String wrapper for `Type`; excluded from XML serialization. | | JobName | string | Optional job name; when not specified, a random value is generated. | | StartUrls | array of string | Initial URLs that start the job. | | Type | [JobTypes](#jobtypes) | Optional job type. | | Headers | [HeadersConfig](#headersconfig) | Optional headers settings. | | Restart | [RestartConfig](#restartconfig) | Optional job restart settings. | | Https | [HttpsConfig](#httpsconfig) | Optional HTTPS settings. | | Cookies | [CookiesConfig](#cookiesconfig) | Optional cookies settings. | | Proxy | [ProxiesConfig](#proxiesconfig) | Optional proxy settings. | | DownloadErrorHandling | [DownloadErrorHandling](#downloaderrorhandling) | Optional download-error handling settings. | | CrawlersProtectionBypass | [CrawlersProtectionBypass](#crawlersprotectionbypass) | Optional crawler-protection countermeasure settings. | | CrossDomainAccess | [CrossDomainAccess](#crossdomainaccess) | Optional cross-domain access settings. | | Retrieval | [RetrievalConfig](#retrievalconfig) | Optional retrieval-index enrollment settings. | | Host | [HostConfig](#hostconfig) | Optional host lifecycle settings. | `AddStartUrl(url)` adds an initial URL. `Parse` accepts the semicolon-separated job configuration; validation requires server and start URLs, then validates configured nested contracts. ### ServerConfig API server configuration. | Name | Type | Description | | --- | --- | --- | | Uri | Uri | API server URI. | `Parse` accepts `wds://user:password@host:port?https=false`, requires the `wds` scheme, and produces an HTTP URI unless `https=true` is supplied. Validation requires a URI. ### JobTypes Defines the environment in which a job crawls. | Name | Description | | --- | --- | | Internet | Crawls internet sources using request gateways. | | Intranet | Crawls intranet sources without those limits. | ### HeadersConfig Configures HTTP headers applied to all job requests. | Name | Type | Description | | --- | --- | --- | | DefaultRequestHeaders | array of [HttpHeader](#httpheader) | Optional headers sent with all requests. | `AddHeader` adds a header; `AppendHeader` adds a value to a named header. `Parse` accepts the configured header fields. ### HttpHeader HTTP header. | Name | Type | Description | | --- | --- | --- | | Name | string | Header name. | | Values | array of string | Header values. | `Parse` accepts `Name: name; Values: value1, value2;`. Validation requires both fields. ### RestartConfig Configures how an existing job is restarted. | Name | Type | Description | | --- | --- | --- | | RestartMode | string | SQL-facing string representation of `JobRestartMode`. | | JobRestartMode | [JobRestartModes](#jobrestartmodes) | Restart mode. | `Parse` accepts `RestartMode: Continue|FromScratch`. ### JobRestartModes Defines the behavior of a restarted job. | Name | Description | | --- | --- | | Continue | Reuses cached data and continues crawling and parsing new data. | | FromScratch | Clears cached data and starts from scratch. | ### HttpsConfig Configures HTTPS certificate validation for target web resources. | Name | Type | Description | | --- | --- | --- | | SuppressHttpsCertificateValidation | bool | When true, suppresses HTTPS certificate validation. | `Parse` accepts `SuppressHttpsCertificateValidation: true|false`. ### CookiesConfig Configures whether the downloader stores response cookies and sends them on later requests. | Name | Type | Description | | --- | --- | --- | | UseCookies | bool | When true, persists and reuses cookies between requests for the job; false is the default. | `Parse` accepts `UseCookies: true|false`. ### ProxiesConfig Configures whether requests use proxies and the pool of available proxy endpoints. | Name | Type | Description | | --- | --- | --- | | UseProxy | bool | Enables proxy use for requests. | | SendOvertRequestsOnProxiesFailure | bool | Enables direct requests when all proxies fail. | | IterateProxyResponseCodes | string | Optional comma-separated HTTP status codes that cause proxy rotation. | | Proxies | array of [ProxyConfig](#proxyconfig) | Optional proxy configurations. | `AddProxyConfig` adds a proxy configuration. `AddProxy` creates and adds one from its protocol, host, port, optional credentials, connection limit, and allowed hosts. ### ProxyConfig Configures one proxy endpoint for download requests. | Name | Type | Description | | --- | --- | --- | | Protocol | string | Proxy protocol (`http`, `https`, or `socks5`). | | Host | string | Proxy host. | | Port | int | Proxy port. | | UserName | string | Optional proxy username. | | Password | string | Optional proxy password. | | ConnectionsLimit | int | Optional maximum concurrent connections through this proxy; null applies no limit. | | AvailableHosts | array of string | Optional lowercase target hosts (or `host:port`) that may use this proxy; null or empty applies no restriction. | | RenewalIntervalSec | int | Optional interval in seconds that caps the wait before this proxy can again be selected for a target host. | `Parse` accepts the semicolon-separated fields shown above. Validation requires `Host` and a nonzero `Port`. `AddAvailableHost(host)` adds a distinct target host. ### DownloadErrorHandling Configures the policy applied when a download fails. | Name | Type | Description | | --- | --- | --- | | ErrorHandlingPolicy | string | SQL-facing string representation of `Policy`; excluded from XML serialization. | | Policy | [DownloadErrorHandlingPolicies](#downloaderrorhandlingpolicies) | Error-handling policy. | | RetryPolicyParams | [RetryPolicyParams](#retrypolicyparams) | Optional retry settings, used when the policy is `Retry`. | `Parse` accepts `ErrorHandlingPolicy: Skip|Retry`. ### DownloadErrorHandlingPolicies Defines how the downloader reacts to an error. | Name | Description | | --- | --- | | Skip | Skips an error and continues crawling. | | Retry | Retries according to the configured retry policy. | ### RetryPolicyParams Retry settings. | Name | Type | Description | | --- | --- | --- | | RetryDelayMs | int | Delay between retries in milliseconds. | | RetriesLimit | int | Maximum number of retries. | `Parse` accepts `RetryDelayMs: 1000; RetriesLimit: 3`. ### CrawlersProtectionBypass Configures per-download response, redirect, timeout, and per-host pacing limits. | Name | Type | Description | | --- | --- | --- | | MaxResponseSizeKb | int | Optional maximum response content buffered for a download; null uses 1000 KB. | | MaxRedirectHops | int | Optional maximum redirects; null uses 10 and `0` disables redirects. | | RequestTimeoutSec | int | Optional request timeout in seconds; null uses 30 seconds. | | CrawlDelays | array of [CrawlDelay](#crawldelay) | Optional per-host delay overrides; null or empty uses robots.txt delay when available, otherwise no extra delay. | `AddCrawlDelay` adds or replaces a host delay. `AddDelay` creates a delay from a host and delay string. Validation applies to all configured delays. ### CrawlDelay Configures request pacing for one target host. | Name | Type | Description | | --- | --- | --- | | Host | string | Lowercase target host, or `host:port` for a non-default port, to which the delay applies. | | Delay | string | Per-host request wait: `0`, fixed seconds, a range such as `1-5`, or `robots` to use the robots.txt crawl-delay when available. | `Parse` accepts `Host: host.com; Delay: 0|1-5|robots`. Validation requires both fields and a supported delay format. ### CrossDomainAccess Configures cross-domain navigation behavior for a job. | Name | Type | Description | | --- | --- | --- | | AccessPolicy | [CrossDomainAccessPolicies](#crossdomainaccesspolicies) | Cross-domain access policy. | `Parse` accepts `AccessPolicy: None|Subdomains|CrossDomains`. Validation requires a defined access policy. ### CrossDomainAccessPolicies Defines the permitted navigation scope for a crawl. | Name | Description | | --- | --- | | None | Allows only the original domain. | | Subdomains | Allows the original domain and its subdomains. | | CrossDomains | Allows navigation to any domain. | ### RetrievalConfig Configures optional enrollment of downloaded page content into the retrieval index. | Name | Type | Description | | --- | --- | --- | | EnrollInIndex | bool | When true, converts successfully downloaded page content to Markdown chunks and enrolls it for retrieval. False is the default; cached pages are skipped unless `Force` is true. | | Force | bool | When true, enrolls cached page content instead of skipping it; it has no effect when `EnrollInIndex` is false. | | MaxTokensPerChunk | int | Optional maximum tokens per indexed chunk; null uses 512. | | ContentScopes | array of [RetrievalContentScope](#retrievalcontentscope) | Optional ordered URL-path patterns and selectors; the first matching scope supplies the selector, otherwise the whole page is enrolled. | | WaitForEnrolled | bool | When true, waits up to one minute per page for snippets to become queryable; false returns after submitting enrollment. | `AddContentScope` appends a scope; the method has no XML summary in the source. Validation validates every configured scope. ### RetrievalContentScope Indexed content scope. | Name | Type | Description | | --- | --- | --- | | PathPattern | string | Source XML documentation is missing for this public member. | | Selector | string | Source XML documentation is missing for this public member. | Validation requires both fields. The source needs XML summaries for `PathPattern` and `Selector` before their behavior can be documented more specifically. ### HostConfig Configures host lifecycle settings such as robots.txt and cookie refresh cadence. | Name | Type | Description | | --- | --- | --- | | ReinitPeriodSec | int | Optional host reinitialization interval in seconds; null uses 604800 seconds (7 days). Reinitialization refreshes robots.txt content and renews cookies. | `Parse` accepts `ReinitPeriodSec: seconds`. ### DownloadTask Represents a download task that can be passed to MSSQL CLR functions. | Name | Type | Description | | --- | --- | --- | | Server | [ServerConfig](#serverconfig) | Server connection associated with the task. | | Id | string | Download task identifier. | | Url | string | Download task URL. | Validation requires `Server`, `Id`, and `Url`. # Crawl Extracts links from the supplied task's downloaded page and creates child download tasks, automatically following pending server-side results. ## Syntax ```sql wds.Crawl(@downloadTask, @selector, @attributeName) ``` ## Arguments | Name | Type | Description | | --- | --- | --- | | @downloadTask | [wds.DownloadTask](#downloadtask) | Supplies the task ID and server connection. A null task returns no rows; a task with an error is returned unchanged. | | @selector | string | Selects link-bearing elements. Use `CSS: `, `XPATH: `, or `*` for all anchor elements. | | @attributeName | string | Optional attribute used to obtain a link. Use `val` for inner text; null or empty uses `href`. | ## Returns `TABLE (Task wds.DownloadTask)` — subsequent [wds.DownloadTask](#downloadtask) values with URLs matched by the selector. ## Data Contracts ### DownloadTask Represents a download task that can be passed to MSSQL CLR functions. | Name | Type | Description | | --- | --- | --- | | Server | [ServerConfig](#serverconfig) | Server connection associated with the task. | | Id | string | Download task identifier. | | Url | string | Download task URL. | Validation requires `Server`, `Id`, and `Url`. ### ServerConfig API server configuration. | Name | Type | Description | | --- | --- | --- | | Uri | Uri | API server URI. | `Parse` accepts `wds://user:password@host:port?https=false`, requires the `wds` scheme, and produces an HTTP URI unless `https=true` is supplied. Validation requires a URI. # ScrapeFirst Extracts the first value matching the selector from the supplied task's downloaded page, automatically following pending server-side results. ## Syntax ```sql wds.ScrapeFirst(@downloadTask, @selector, @attributeName) ``` ## Arguments | Name | Type | Description | | --- | --- | --- | | @downloadTask | [wds.DownloadTask](#downloadtask) | Supplies the task ID and server connection. A null task produces no value; a task with an error produces that error as a value. | | @selector | string | Selects matching elements. Use `CSS: `, `XPATH: `, or `*` for the whole page. | | @attributeName | string | Optional matched-element attribute. Use `val`, null, or empty for inner text. | ## Returns `NVARCHAR(MAX)` — the first scraped value. An unmatched selector returns `NULL`. ## Data Contracts ### DownloadTask Represents a download task that can be passed to MSSQL CLR functions. | Name | Type | Description | | --- | --- | --- | | Server | [ServerConfig](#serverconfig) | Server connection associated with the task. | | Id | string | Download task identifier. | | Url | string | Download task URL. | Validation requires `Server`, `Id`, and `Url`. ### ServerConfig API server configuration. | Name | Type | Description | | --- | --- | --- | | Uri | Uri | API server URI. | `Parse` accepts `wds://user:password@host:port?https=false`, requires the `wds` scheme, and produces an HTTP URI unless `https=true` is supplied. Validation requires a URI. # ScrapeAll Extracts every value matching the selector from the supplied task's downloaded page, automatically following pending server-side results. ## Syntax ```sql wds.ScrapeAll(@downloadTask, @selector, @attributeName) ``` ## Arguments | Name | Type | Description | | --- | --- | --- | | @downloadTask | [wds.DownloadTask](#downloadtask) | Supplies the task ID and server connection. A null task produces no values; a task with an error produces that error as a value. | | @selector | string | Selects matching elements. Use `CSS: `, `XPATH: `, or `*` for the whole page. | | @attributeName | string | Optional matched-element attribute. Use `val`, null, or empty for inner text. | ## Returns `TABLE (Data NVARCHAR(MAX))` — one `Data` row for every matched page element. ## Data Contracts ### DownloadTask Represents a download task that can be passed to MSSQL CLR functions. | Name | Type | Description | | --- | --- | --- | | Server | [ServerConfig](#serverconfig) | Server connection associated with the task. | | Id | string | Download task identifier. | | Url | string | Download task URL. | Validation requires `Server`, `Id`, and `Url`. ### ServerConfig API server configuration. | Name | Type | Description | | --- | --- | --- | | Uri | Uri | API server URI. | `Parse` accepts `wds://user:password@host:port?https=false`, requires the `wds` scheme, and produces an HTTP URI unless `https=true` is supplied. Validation requires a URI. # ScrapeMultiple Creates a fluent batch-scrape builder bound to a download task. The server request is sent only when the builder retrieves values. ## Syntax ```sql wds.ScrapeMultiple(@downloadTask) ``` ## Arguments | Name | Type | Description | | --- | --- | --- | | @downloadTask | [wds.DownloadTask](#downloadtask) | Supplies the task used by the later batch request. A null task remains unbound and later retrieval produces no values. | ## Returns [wds.ScrapeMultipleParams](#scrapemultipleparams) — use its `AddScrapeParams`, `GetAll`, and `GetFirst` methods to configure and retrieve the batch result. See [Scrape Paged](../examples/scrape-paged.html) for a complete `ScrapeMultiple(...).AddScrapeParams(...).GetFirst(...)` workflow. ## Data Contracts ### DownloadTask Represents a download task that can be passed to MSSQL CLR functions. | Name | Type | Description | | --- | --- | --- | | Server | [ServerConfig](#serverconfig) | Server connection associated with the task. | | Id | string | Download task identifier. | | Url | string | Download task URL. | Validation requires `Server`, `Id`, and `Url`. ### ServerConfig API server configuration. | Name | Type | Description | | --- | --- | --- | | Uri | Uri | API server URI. | `Parse` accepts `wds://user:password@host:port?https=false`, requires the `wds` scheme, and produces an HTTP URI unless `https=true` is supplied. Validation requires a URI. ### ScrapeMultipleParams A special object for fluent configuration of a batch scrape request. | Name | Type | Description | | --- | --- | --- | `BindDownloadTask(downloadTask)` binds the parent task. `AddScrapeParams(name, selector, attributeName)` adds an extraction rule. `GetAll(name)` returns [StringDataItems](#stringdataitems), and `GetFirst(name)` returns the first scraped string. `GetFirst(name)` returns `NULL` when no value is available. Validation requires a bound task and validates all configured scrape parameters. ### ScrapeParams Defines one named field extracted from pages by selector, attribute, and optional conversion. | Name | Type | Description | | --- | --- | --- | | Name | string | Output field name; names must be unique within a scrape request or Traversal level. | | Selector | string | Selector for matching elements: `CSS: `, `XPATH: `, or `*` for the whole page. | | AttributeName | string | Optional attribute; `val`, null, or empty returns inner text. | | Convert | string | Optional conversion: `md()` for Markdown or `sr()` for main readable content. | Validation requires `Name` and `Selector`. ### ScrapeResult Represents values extracted for one named scrape field. | Name | Type | Description | | --- | --- | --- | | Name | string | Optional field name copied from the matching `ScrapeParams` entry. | | Values | array of string | Scraped values. | Validation requires `Name` and `Values`. ### StringDataItems A special object for the `ToStringsTable` function. | Name | Type | Description | | --- | --- | --- | | DataItems | array of string | Source XML documentation is missing for this public field. | `AddDataItems(items)` adds data items. `Parse` is unsupported. # TaskStatus Reads the supplied task's download state and HTTP attempt history from the MSSQL-compatible API. ## Syntax ```sql wds.TaskStatus(@downloadTask) ``` ## Arguments | Name | Type | Description | | --- | --- | --- | | @downloadTask | [wds.DownloadTask](#downloadtask) | Supplies the download task ID and server connection. A null task returns a null status; a task with an error returns that error without calling the server. | ## Returns [wds.DownloadTaskStatus](#downloadtaskstatus) — a task-status object. ## Data Contracts ### DownloadTask Represents a download task that can be passed to MSSQL CLR functions. | Name | Type | Description | | --- | --- | --- | | Server | [ServerConfig](#serverconfig) | Server connection associated with the task. | | Id | string | Download task identifier. | | Url | string | Download task URL. | Validation requires `Server`, `Id`, and `Url`. ### ServerConfig API server configuration. | Name | Type | Description | | --- | --- | --- | | Uri | Uri | API server URI. | `Parse` accepts `wds://user:password@host:port?https=false`, requires the `wds` scheme, and produces an HTTP URI unless `https=true` is supplied. Validation requires a URI. ### DownloadTaskStatus Reports the current state and request history for a download task. | Name | Type | Description | | --- | --- | --- | | Url | string | Downloaded URL. | | TaskState | string | String wrapper for `State`; it is excluded from XML serialization. | | State | [DownloadTaskStates](#downloadtaskstates) | Task state. | | Result | [DownloadInfo](#downloadinfo) | Optional final download-attempt details. | | IntermedResults | array of [DownloadInfo](#downloadinfo) | Optional earlier attempt details, such as redirects or proxy retries. | ### DownloadTaskStates Defines the lifecycle state reported for a download task. | Name | Description | | --- | --- | | Handled | The downloader finished the task; inspect `Result.IsSuccess` and `Result.HttpStatusCode` when a result is present. | | AccessDeniedForRobots | Access to the URL is denied by robots.txt. | | AllRequestGatesExhausted | All request gateways were exhausted without receiving data. | | Created | Task has not started. | | InProgress | Task is in progress. | | Deleted | Task has been deleted. | ### DownloadInfo Download attempt information. | Name | Type | Description | | --- | --- | --- | | Method | string | HTTP method. | | Url | string | Request URL. | | IsSuccess | bool | Indicates whether the request succeeded. | | HttpStatusCode | int | HTTP status code. | | ReasonPhrase | string | HTTP reason phrase. | | RequestHeaders | array of [HttpHeader](#httpheader) | HTTP headers sent with the request. | | ResponseHeaders | array of [HttpHeader](#httpheader) | HTTP headers received in the response. | | RequestCookies | array of [Cookie](#cookie) | Cookies sent with the request. | | ResponseCookies | array of [Cookie](#cookie) | Cookies received in the response. | | RequestDateUtc | DateTime | Request date and time in UTC. | | DownloadTimeSec | double | Download time in seconds. | | ViaProxy | bool | Indicates whether the request was made through a proxy. | | WaitTimeSec | double | Delay in seconds before the request was executed. | | CrawlDelaySec | int | Delay in seconds applied to the request. | Validation requires `Method` and `Url`. ### HttpHeader HTTP header. | Name | Type | Description | | --- | --- | --- | | Name | string | Header name. | | Values | array of string | Header values. | `Parse` accepts `Name: name; Values: value1, value2;`. Validation requires both fields. ### Cookie Represents an HTTP cookie observed during a download request or response. | Name | Type | Description | | --- | --- | --- | | Name | string | Cookie name. | | Value | string | Optional cookie value. | | Domain | string | Optional cookie domain attribute. | | Path | string | Optional cookie path attribute. | | HttpOnly | bool | `HttpOnly` cookie attribute. | | Secure | bool | `Secure` cookie attribute. | | Expires | DateTime | Optional expiration date and time; null for a session cookie or an unknown expiration. | Validation requires `Name`. # ToStringsTable Projects supplied string items into one table row per value for use with `ScrapeMultipleParams.GetAll`. ## Syntax ```sql wds.ToStringsTable(@items) ``` ## Arguments | Name | Type | Description | | --- | --- | --- | | @items | [wds.StringDataItems](#stringdataitems) | Supplies the string items to project into table rows. | ## Returns `TABLE (Data NVARCHAR(MAX))` — the input-items table. ## Data Contracts ### StringDataItems A special object for the `ToStringsTable` function. | Name | Type | Description | | --- | --- | --- | | DataItems | array of string | Source XML documentation is missing for this public field. | `AddDataItems(items)` adds data items. `Parse` is unsupported. # WDS for MS SQL Server Bring web crawling and scraping into T‑SQL. WDS for MS SQL Server is a CLR library with user‑defined types (UDTs) and functions that let you start jobs, discover pages, extract data, and check task status directly from SQL. See [Plans](https://webdatasource.com/pricing.html) for feature availability. ## What You Can Do - Start: launch a job with a `JobConfig` and receive initial `DownloadTask`s. - Crawl: discover follow‑up pages and get new `DownloadTask`s. - Scrape: extract one value (ScrapeFirst), all values (ScrapeAll), or multiple fields in one call (ScrapeMultiple). - Inspect: query `DownloadTaskStatus` to monitor progress and debug issues. ## Prerequisites - SQL Server: tested with SQL Server 2022 (Windows). - WDS API Server: running and reachable (see [Deployments](../server/deployments/index.html)). ## Components - CLR Functions: ServerStatus, Start, Crawl, ScrapeFirst, ScrapeAll, ScrapeMultiple, TaskStatus, and ToStringsTable — see [CLR Functions](./clr-functions/index.html). - UDT contracts: 23 installed SQL types configure jobs and carry results. Their source-backed descriptions are embedded in the function pages that use them; start with [JobConfig](./clr-functions/start.html#jobconfig), [DownloadTask](./clr-functions/crawl.html#downloadtask), and [DownloadTaskStatus](./clr-functions/task-status.html#downloadtaskstatus). ## Examples Explore end‑to‑end scripts for common scenarios — see [Examples](./examples/index.html) ## Install Step‑by‑step instructions to enable CLR and load the library — see [Install](./clr-functions/install.html) ## Support If you hit issues, please use [GitHub Issues](https://github.com/webdatasource/wds.mssql.clr/issues). # Examples # Scripts Examples This section contains T-SQL script examples that might be used for evaluation or as a base for writing queries to real web sources.\ All these examples use the [Playground auxiliary container](../../server/deployments/dockercompose.html#auxiliary-containers) as the target web resource. By default, the RestartMode is [Continue](../clr-functions/start.html#jobrestartmodes), so the first runs would take some time to grab real data from the playground, but the rest will work on the cache, which is much faster. If you don't like this, [change the RestartMode](../clr-functions/start.html#restartconfig). - [Scrape Paged](/releases/latest/mssql/examples/scrape-paged.html) - [Scrape Sitemap](/releases/latest/mssql/examples/scrape-sitemap.html) # Scrape Paged > **_Minimal playground version:_** v1.0.0 Demonstrates iterating through paginated category pages to visit item pages and scrape fields; includes a ScrapeMultiple variant for fewer API calls. ``` SQL DECLARE @jobConfig wds.JobConfig = 'JobName: CrawlAllProducts; Server: wds://localhost:2807; StartUrls: http://playground'; DECLARE @pages TABLE (Task wds.DownloadTask); -- Gathering categories' first pages INSERT INTO @pages (Task) SELECT nav.Task FROM wds.Start(@jobConfig) root OUTER APPLY wds.Crawl(root.Task, 'css: ul.nav a:not([href="/"])', null) nav -- Gathering the other pages of categories WHILE @@ROWCOUNT > 0 BEGIN INSERT INTO @pages (Task) SELECT newPages.Task FROM @pages curPages CROSS APPLY wds.Crawl(curPages.Task, 'css: ul.pagination li:not(.disabled) a', null) newPages WHERE NOT EXISTS (SELECT NULL FROM @pages p WHERE p.Task = newPages.Task) END -- Iterating through pages, visiting product pages, and scraping data from them SELECT products.Task.Url ProductUrl, wds.ScrapeFirst(products.Task, 'css: h1', null) AS ProductName, wds.ScrapeFirst(products.Task, 'css: .price span', null) AS ProductPrice FROM @pages pages CROSS APPLY wds.Crawl(pages.Task, 'css: .table a', null) products ``` Iterate over all pages and scrape data using the [ScrapeMultiple](../clr-functions/scrape-multiple.html) function. This approach is a bit faster because fewer requests to the WDS API Server are required. ``` SQL DECLARE @jobConfig wds.JobConfig = 'JobName: CrawlAllProductsBatch; Server: wds://localhost:2807; StartUrls: http://playground'; DECLARE @pages TABLE (Task wds.DownloadTask); -- Gathering categories' first pages INSERT INTO @pages (Task) SELECT nav.Task FROM wds.Start(@jobConfig) root OUTER APPLY wds.Crawl(root.Task, 'css: ul.nav a:not([href="/"])', null) nav -- Gathering the other pages of categories WHILE @@ROWCOUNT > 0 BEGIN INSERT INTO @pages (Task) SELECT newPages.Task FROM @pages curPages CROSS APPLY wds.Crawl(curPages.Task, 'css: ul.pagination li:not(.disabled) a', null) newPages WHERE NOT EXISTS (SELECT NULL FROM @pages p WHERE p.Task = newPages.Task) END -- Iterating through pages, visiting product pages, and scraping data from them SELECT products.Task.Url ProductUrl, product.ScrapeResult.GetFirst('ProductName') AS ProductName, product.ScrapeResult.GetFirst('ProductPrice') AS ProductPrice FROM @pages pages CROSS APPLY wds.Crawl(pages.Task, 'css: .table a', null) products CROSS APPLY ( SELECT wds.ScrapeMultiple(products.Task) .AddScrapeParams('ProductName', 'css: h1', null) .AddScrapeParams('ProductPrice', 'css: .price span', null) AS ScrapeResult ) product ``` # Scrape Sitemap > **_Minimal playground version:_** v1.0.1 Shows how to collect product pages from sitemap.xml and scrape fields from each page; includes a ScrapeMultiple variant to minimize round trips. ``` SQL DECLARE @jobConfig wds.JobConfig = 'JobName: CrawlAllProductsSitemap; Server: wds://localhost:2807; StartUrls: http://playground/sitemap.xml'; SELECT products.Task.Url ProductUrl, wds.ScrapeFirst(products.Task, 'css: h1', null) AS ProductName, wds.ScrapeFirst(products.Task, 'css: .price span', null) AS ProductPrice FROM wds.Start(@jobConfig) root OUTER APPLY wds.Crawl(root.Task, 'xpath: //*[local-name()="url"]/*[local-name()="loc"]', 'val') products ``` Getting product pages from a sitemap.xml and scrape data from these pages using the [ScrapeMultiple](../clr-functions/scrape-multiple.html) function. This approach is a bit faster because fewer requests to the WDS API Server are required. ``` SQL DECLARE @jobConfig wds.JobConfig = 'JobName: CrawlAllProductsSitemap; Server: wds://localhost:2807; StartUrls: http://playground/sitemap.xml'; SELECT products.Task.Url ProductUrl, product.ScrapeResult.GetFirst('ProductName') AS ProductName, product.ScrapeResult.GetFirst('ProductPrice') AS ProductPrice FROM wds.Start(@jobConfig) root OUTER APPLY wds.Crawl(root.Task, 'xpath: //*[local-name()="url"]/*[local-name()="loc"]', 'val') products CROSS APPLY ( SELECT wds.ScrapeMultiple(products.Task) .AddScrapeParams('ProductName', 'css: h1', null) .AddScrapeParams('ProductPrice', 'css: .price span', null) AS ScrapeResult ) product ``` # MCP Server # MCP Server The WDS API Server exposes a Model Context Protocol endpoint for IDEs and agentic systems. Its current surface consists of 21 registered tools for job, download, Traversal, scheduling, tenant, and retrieval workflows, plus seven registered prompts that compose those tools into common tasks. See [Plans](https://webdatasource.com/pricing.html) for feature availability. ## Connect to the MCP Server Deploy the [WDS API Server](../server/index.html), then connect an MCP client to its HTTP endpoint. | Parameter | Value | Description | | --- | --- | --- | | Name | `wds` | Example client registration name used by the prompt commands in this documentation | | Transport | Streamable HTTP | MCP transport exposed by the API Server | | URL | `http://[host:port]/mcp` | Use `http://localhost:2807/mcp` for the default local Docker port | A deployment mounted below a base path must prepend that path to `/mcp`. For example, a base path of `/wds` produces `/wds/mcp`. For Visual Studio Code setup, see its [MCP server documentation](https://code.visualstudio.com/docs/copilot/chat/mcp-servers). ## Tools The [MCP Tools catalog](./tools/index.html) documents all registered names, arguments, defaults, return schemas, and behavior flags. Current capabilities include: - Manage jobs with [WdsGetJobs](./tools/get-jobs-info.html), [WdsGetJobConfig](./tools/get-job-config.html), [WdsUpsertJobConfig](./tools/upsert-job-config.html), and [WdsDeleteJob](./tools/delete-job.html). - Fetch one page with [WdsFetchChunked](./tools/fetch-chunked.html), then poll and read its content with [WdsGetChunk](./tools/get-chunk.html). - Store and run hierarchical plans with [WdsUpsertTraversalConfig](./tools/upsert-traversal-config.html), [WdsRunTraversal](./tools/run-traversal.html), and the Traversal inspection tools. - Configure recurring runs with the three `Wds*TraversalSchedule` tools when Scheduling is enabled. - Search retrieval-indexed content with [WdsRetrieve](./tools/retrieve.html) when Retrieval is enabled. - Remove all data for the default MCP tenant with [WdsDeleteTenant](./tools/delete-tenant.html). ## Prompts The [MCP Prompts catalog](./prompts/index.html) documents the seven exact registered prompt names: - [`assess`](./prompts/assess.html) proposes a Traversal extraction plan; [`scrape`](./prompts/scrape.html) runs an already configured plan and returns its data. - [`resume`](./prompts/resume.html) and [`sliced-resume`](./prompts/sliced-resume.html) crawl and summarize a site. - [`index`](./prompts/index-wr.html) and [`reindex`](./prompts/reindex-wr.html) enroll content for retrieval; [`query`](./prompts/query.html) answers a task from that index. ## Typical Flows ### Fetch One Page 1. Persist a job with `WdsUpsertJobConfig`. 2. Call `WdsFetchChunked` with the job name and absolute URL. 3. Poll `WdsGetChunk` with the returned task ID until content is available. 4. Continue reading spans with `startIndex` and `length` when the page is larger than one response. ### Run a Traversal 1. Persist `JobConfig` with `WdsUpsertJobConfig`. 2. Persist `TraversalConfig` with `WdsUpsertTraversalConfig`. 3. Start the run with `WdsRunTraversal` and retain its `runNum`. 4. Poll `WdsGetTraversalRunInfo`, inspect `WdsGetTraversalRunErrors`, and page through `WdsGetTraversalRunData` until `dataCursor` is null. ### Use Retrieval Prompts 1. Invoke `index`, or `reindex` when cached content must be reenrolled. 2. After enrollment completes, invoke `query` with the target URL and task. # WDS MCP Prompts # WDS MCP Prompts WDS supplies seven registered MCP prompts that guide an AI agent through site assessment, prepared Traversal extraction, site summaries, and retrieval-index workflows. Each prompt page documents the exact call arguments and the complete prompt text returned to the client. ## Prompt Catalog - [`assess`](./assess.html) — Assess a web resource and propose a traversal config for data extraction. - [`scrape`](./scrape.html) — Run a configured traversal job and return gathered data. - [`resume`](./resume.html) — Crawl a web resource and prepare a resume of what it is about and what it offers. - [`sliced-resume`](./sliced-resume.html) — Crawl a web resource by slices and build a resume from the currently available slices before continuing through the remaining site. - [`index`](./index-wr.html) — Describes to an AI agent how to enroll a web resource into an index for further retrieval. - [`reindex`](./reindex-wr.html) — Describes to an AI agent how to reenroll a web resource into an index for further retrieval. - [`query`](./query.html) — Answers a question based on the data available in the WDS MCP Server. ## Typical Workflows - Use `assess` to inspect a site and propose a `TraversalConfig`. After creating the job and storing that config with the MCP tools, use `scrape` to run it and return all cursor-paginated data. - Use `resume` for a complete site summary or `sliced-resume` when an initial partial summary should be shown before the remaining crawl continues. - Use `index` for the first retrieval enrollment, `reindex` to force reenrollment, and `query` to answer a task from the indexed content. ## How to Run Ensure the [MCP Server](../index.html) is connected in your client, then invoke the registered prompt name. Each prompt page shows the client command, arguments, defaults, and returned text. ## Table of Contents - [resume](/releases/latest/mcp/prompts/resume.html) - [sliced-resume](/releases/latest/mcp/prompts/sliced-resume.html) - [assess](/releases/latest/mcp/prompts/assess.html) - [scrape](/releases/latest/mcp/prompts/scrape.html) - [index](/releases/latest/mcp/prompts/index-wr.html) - [reindex](/releases/latest/mcp/prompts/reindex-wr.html) - [query](/releases/latest/mcp/prompts/query.html) # resume Crawl a web resource and prepare a resume of what it is about and what it offers. ## How to call In supported clients the command syntax can vary. If the WDS MCP server is registered as `wds`, use: | Client | Command | | --- | --- | | Visual Studio Code | `/mcp.wds.resume` | ## Arguments | Name | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | url | string | No | `http://playground` | A website URL | ## Returned Prompt Text The block below contains the complete prompt returned to the MCP client. Runtime placeholders map to arguments as follows: - `` — `url` ````nohighlight ROLE: Act as a web developer with experience in designing multi-page web resources, including paged data, templating, and expertise in pure CSS as well as in modern CSS frameworks: Bootstrap, Tailwind CSS, Foundation, Bulma, UIkit, Tachyons, Materialize, Metro, Spectre.css, Milligram, Pure.css, Skeleton, NES.css, W3.CSS, Picnic CSS, Basscss, Shoelace, etc. Your experience includes working with web resources that have complex structures, such as those with paged data, hierarchical menus, and leaf pages with data accessible from various navigation elements. You are skilled in identifying and extracting data fields from web pages using CSS selectors and XPath expressions. OBJECTIVE: Crawl through all pages of TARGET_URL website using WDS MCP tools, gather its content, and prepare a resume of what the website is about and what it offers. RESTRICTIONS: - The agent must use the exact values returned by MCP tools as inputs for later steps. Do not rewrite or guess job names, URLs, run numbers, cursors, or selectors. - MCP structured tool results are serialized as lower-camel JSON fields. Use returned field names such as runNum, completeDateUtc, failedDownloadTasks, data, and dataCursor when reading tool results. - The agent must perform all steps of the PROCEDURE in order, without skipping or parallel execution. - Call MCP tools with named arguments matching the tool parameter names. - WdsFetchChunked returns the download task ID as a string. It does not return raw page text. - Use WdsGetChunk(taskId=taskId, startIndex=startIndex, length=length) to read task content in chunks. Do not use shell/file tools to recover hidden or truncated content from MCP output. - Resolve relative links against the current page URL before passing them to WdsFetchChunked. - To change a JobConfig, read the current object with WdsGetJobConfig when it exists, create or modify the complete object in memory, then persist it with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg). Preserve all unrelated existing properties. - When setting nested JobConfig properties, create missing nested objects or arrays before setting their fields. - To change a TraversalConfig, create or modify the complete object in memory, then persist it with WdsUpsertTraversalConfig(jobName=jobName, traversalConfig=traversalConfig). - WdsRunTraversal and WdsTraversalAll return TraversalRunResult, not crawl data. Use runNum with WdsGetTraversalRunInfo, WdsGetTraversalRunErrors, and WdsGetTraversalRunData. - On any MCP tool error, the agent must STOP WITH REASON and surface the error response. - Do not stop reading gathered data while dataCursor is not null. GLOBAL CONSTANTS: TARGET_URL = '' PROCEDURE (EXECUTE IN EXACT ORDER) 1. Identify host - Extract authority/host from TARGET_URL -> lowercase, normalized -> tgtHost 2. Get or create job - jobs = WdsGetJobs() - Find job where job.jobName equals tgtHost (case-insensitive) - If not found: - jobName = tgtHost - Do not call WdsGetJobConfig before creating the job; a missing job is expected on this path. - Determine jobType: - 'Intranet' if host is single-label or ends with .local, .internal, .svc.cluster.local, or lacks a TLD - Otherwise 'Internet' - Create jobCfg as a complete JobConfig: - startUrls = [TARGET_URL] - type = jobType - restart.jobRestartMode = 'Continue' - Persist jobCfg with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg) - jobCfg = WdsGetJobConfig(jobName=jobName) - If found: - jobName = job.jobName - jobCfg = WdsGetJobConfig(jobName=jobName) 3. Configure job - Update jobCfg in memory, preserving all unrelated properties: - jobCfg.restart.jobRestartMode = 'Continue' - Persist jobCfg with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg) - jobCfg = WdsGetJobConfig(jobName=jobName) 4. Start crawling - convert = 'md()'; - maxDepth = null - traversalRunResult = WdsTraversalAll(jobName=jobName, convert=convert, maxDepth=maxDepth) - runNum = traversalRunResult.runNum - Do not infer runNum from job state, run lists, previous output, or arithmetic. 5. Validate crawl completion - Wait for the traversal run to complete: - maxPolls = 60 - pollIntervalSec = 1 - pollCount = 0 - lastTraversalRunInfo = null - REPEAT while pollCount < maxPolls: - pollCount = pollCount + 1 - traversalRunInfo = WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) - lastTraversalRunInfo = traversalRunInfo - If traversalRunInfo.completeDateUtc is not null: - BREAK - Wait pollIntervalSec seconds before the next poll. - Continue polling with WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) until traversalRunInfo.completeDateUtc is not null or pollCount reaches maxPolls. - If lastTraversalRunInfo.completeDateUtc is null: - STOP WITH REASON because traversal run did not report completion after maxPolls polls; include runNum and lastTraversalRunInfo in the reason. - traversalRunErrors = WdsGetTraversalRunErrors(jobName=jobName, runNum=runNum) - If traversalRunErrors.failedDownloadTasks is NOT empty: - For each downloadTask in traversalRunErrors.failedDownloadTasks: - For each failedDownloadTask in downloadTask.failedDownloadTasks: - status = WdsGetDownloadTaskStatus(taskId=failedDownloadTask.id) - If status.result is null: - Collect status.url, null, null - If status.result is not null: - Collect status.url, status.result.httpStatusCode, status.result.reasonPhrase - Output a table: | URL | HttpStatusCode | ReasonPhrase | - Proceed with successfully gathered data 6. Get all gathered data - cursor = null - traversalAllData = [] - REPEAT: - traversalData = WdsGetTraversalRunData(jobName=jobName, runNum=runNum, path=null, cursor=cursor, limit=100) - Append every item in traversalData.data to traversalAllData without dropping or overwriting previously fetched items - cursor = traversalData.dataCursor - UNTIL cursor is null 7. Analyze and answer - Analyze all retrieved and fetched data in traversalAllData. - Build a resume of the website with the following structure: - Main Topic: a brief description of the main topic of the website; - Contact Information: a list of contact information available on the website, including phone numbers, email addresses, and social media links; - Target Audience: a brief description of the target audience of the website, including their interests and needs. - Services: a list of services found on the website (if they are), with a full description of each service. - Products: a list of products found on the website (if they are), with a full description of each product; - FAQ: a list of frequently asked questions found on the website, with answers to each question; ```` # sliced-resume Crawl a web resource by slices and build a resume of what it is about and what it offers based on the currently available slices. ## How to call In supported clients the command syntax can vary. If the WDS MCP server is registered as `wds`, use: | Client | Command | | --- | --- | | Visual Studio Code | `/mcp.wds.sliced-resume` | ## Arguments | Name | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | url | string | No | `http://playground` | A website URL | ## Returned Prompt Text The block below contains the complete prompt returned to the MCP client. Runtime placeholders map to arguments as follows: - `` — `url` ````nohighlight ROLE: Act as a web developer with experience in designing multi-page web resources, including paged data, templating, and expertise in pure CSS as well as in modern CSS frameworks: Bootstrap, Tailwind CSS, Foundation, Bulma, UIkit, Tachyons, Materialize, Metro, Spectre.css, Milligram, Pure.css, Skeleton, NES.css, W3.CSS, Picnic CSS, Basscss, Shoelace, etc. Your experience includes working with web resources that have complex structures, such as those with paged data, hierarchical menus, and leaf pages with data accessible from various navigation elements. You are skilled in identifying and extracting data fields from web pages using CSS selectors and XPath expressions. OBJECTIVE: Crawl through all pages of TARGET_URL website using WDS MCP tools, gather its content, and prepare a resume of what the website is about and what it offers. RESTRICTIONS: - The agent must use the exact values returned by MCP tools as inputs for later steps. Do not rewrite or guess job names, URLs, run numbers, cursors, or selectors. - MCP structured tool results are serialized as lower-camel JSON fields. Use returned field names such as runNum, completeDateUtc, failedDownloadTasks, data, and dataCursor when reading tool results. - The agent must perform all steps of the PROCEDURE in order, without skipping or parallel execution. - Call MCP tools with named arguments matching the tool parameter names. - WdsFetchChunked returns the download task ID as a string. It does not return raw page text. - Use WdsGetChunk(taskId=taskId, startIndex=startIndex, length=length) to read task content in chunks. Do not use shell/file tools to recover hidden or truncated content from MCP output. - Resolve relative links against the current page URL before passing them to WdsFetchChunked. - To change a JobConfig, read the current object with WdsGetJobConfig when it exists, create or modify the complete object in memory, then persist it with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg). Preserve all unrelated existing properties. - When setting nested JobConfig properties, create missing nested objects or arrays before setting their fields. - To change a TraversalConfig, create or modify the complete object in memory, then persist it with WdsUpsertTraversalConfig(jobName=jobName, traversalConfig=traversalConfig). - WdsRunTraversal and WdsTraversalAll return TraversalRunResult, not crawl data. Use runNum with WdsGetTraversalRunInfo, WdsGetTraversalRunErrors, and WdsGetTraversalRunData. - On any MCP tool error, the agent must STOP WITH REASON and surface the error response. GLOBAL CONSTANTS: TARGET_URL = '' CONVERT = 'md()' PROCEDURE (EXECUTE IN EXACT ORDER) 1. Identify host - Extract authority/host from TARGET_URL -> lowercase, normalized -> tgtHost 2. Get or create job - jobs = WdsGetJobs() - Find job where job.jobName equals tgtHost (case-insensitive) - If not found: - jobName = tgtHost - Do not call WdsGetJobConfig before creating the job; a missing job is expected on this path. - Determine jobType: - 'Intranet' if host is single-label or ends with .local, .internal, .svc.cluster.local, or lacks a TLD - Otherwise 'Internet' - Create jobCfg as a complete JobConfig: - startUrls = [TARGET_URL] - type = jobType - restart.jobRestartMode = 'Continue' - Persist jobCfg with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg) - jobCfg = WdsGetJobConfig(jobName=jobName) - If found: - jobName = job.jobName - jobCfg = WdsGetJobConfig(jobName=jobName) 3. Configure job - Update jobCfg in memory, preserving all unrelated properties: - jobCfg.restart.jobRestartMode = 'Continue' - Persist jobCfg with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg) - jobCfg = WdsGetJobConfig(jobName=jobName) 4. Start crawling - convert = CONVERT - maxDepth = 0 - traversalRunResult = WdsTraversalAll(jobName=jobName, convert=convert, maxDepth=maxDepth) - runNum = traversalRunResult.runNum - Do not infer runNum from job state, run lists, previous output, or arithmetic. 5. Validate crawl completion - Wait for the traversal run to complete: - maxPolls = 60 - pollIntervalSec = 1 - pollCount = 0 - lastTraversalRunInfo = null - REPEAT while pollCount < maxPolls: - pollCount = pollCount + 1 - traversalRunInfo = WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) - lastTraversalRunInfo = traversalRunInfo - If traversalRunInfo.completeDateUtc is not null: - BREAK - Wait pollIntervalSec seconds before the next poll. - Continue polling with WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) until traversalRunInfo.completeDateUtc is not null or pollCount reaches maxPolls. - If lastTraversalRunInfo.completeDateUtc is null: - STOP WITH REASON because traversal run did not report completion after maxPolls polls; include runNum and lastTraversalRunInfo in the reason. - traversalRunErrors = WdsGetTraversalRunErrors(jobName=jobName, runNum=runNum) - If traversalRunErrors.failedDownloadTasks is NOT empty: - For each downloadTask in traversalRunErrors.failedDownloadTasks: - For each failedDownloadTask in downloadTask.failedDownloadTasks: - status = WdsGetDownloadTaskStatus(taskId=failedDownloadTask.id) - If status.result is null: - Collect status.url, null, null - If status.result is not null: - Collect status.url, status.result.httpStatusCode, status.result.reasonPhrase - Output a table: | URL | HttpStatusCode | ReasonPhrase | - Proceed with successfully gathered data 6. Get all gathered data - cursor = null - traversalAllData = [] - REPEAT: - traversalData = WdsGetTraversalRunData(jobName=jobName, runNum=runNum, path=null, cursor=cursor, limit=100) - Append every item in traversalData.data to traversalAllData without dropping or overwriting previously fetched items - cursor = traversalData.dataCursor - UNTIL cursor is null 7. Analyze the first part and answer - Analyze all retrieved and fetched data in traversalAllData. - Build the first part of a resume of the website with the following structure: - Main Topic: a brief description of the main topic of the website; - Contact Information: a list of contact information available on the website, including phone numbers, email addresses, and social media links; - FAQ: a list of frequently asked questions found on the website, with answers to each question; - SHOW the partial resume obtained so far to the user. 8. Continue crawling the rest of the data without confirmation. - Inform users that you are continuing crawling the rest of the data automatically. - Do not ask users for confirmation, just continue automatically. 9. Check robots.txt and inform user about CrawlDelay if any - robotsUrl = scheme and authority from TARGET_URL + '/robots.txt' - Read robots.txt content: - taskId = WdsFetchChunked(jobName=jobName, url=robotsUrl) - REPEAT: - contentResult = WdsGetChunk(taskId=taskId, startIndex=0, length=1024) - If contentResult is not null: BREAK - Continue polling WdsGetChunk with the same arguments until contentResult is not null. - robots = contentResult.contentSpan if it is not null; otherwise '' - contentLength = contentResult.contentLength if it is not null; otherwise length(robots) - If more unseen content is required for the current analysis: - REPEAT: - startIndex = length(robots) - If startIndex >= contentLength: BREAK - contentResult = WdsGetChunk(taskId=taskId, startIndex=startIndex, length=1024) - If contentResult is null: continue polling WdsGetChunk with the same arguments until contentResult is not null - Append contentResult.contentSpan to robots - UNTIL enough relevant content has been inspected or startIndex >= contentLength - Analyze only the content present in robots. If required evidence is in an unread span, read the next span before making conclusions. - If robots is not null and robots is not empty: - Parse robots as plain text. Find the first case-insensitive 'Crawl-delay:' directive and set crawlDelay to its trimmed value. - If crawlDelay is not null: - SHOW user the Crawl-delay found in robots.txt: crawlDelay seconds. - Update jobCfg in memory, preserving all unrelated properties: - If crawlDelay is not null, ensure jobCfg.crawlersProtectionBypass.crawlDelays contains { host: tgtHost, delay: crawlDelay } - If crawlDelay is null, ensure jobCfg.crawlersProtectionBypass.crawlDelays contains { host: tgtHost, delay: 'robots' } - jobCfg.restart.jobRestartMode = 'Continue' - Persist jobCfg with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg) 10. Continue crawling - maxDepth = null - traversalRunResult = WdsTraversalAll(jobName=jobName, convert=convert, maxDepth=maxDepth) - runNum = traversalRunResult.runNum - Do not infer runNum from job state, run lists, previous output, or arithmetic. 11. Validate crawl completion - Wait for the traversal run to complete: - maxPolls = 60 - pollIntervalSec = 1 - pollCount = 0 - lastTraversalRunInfo = null - REPEAT while pollCount < maxPolls: - pollCount = pollCount + 1 - traversalRunInfo = WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) - lastTraversalRunInfo = traversalRunInfo - If traversalRunInfo.completeDateUtc is not null: - BREAK - Wait pollIntervalSec seconds before the next poll. - Continue polling with WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) until traversalRunInfo.completeDateUtc is not null or pollCount reaches maxPolls. - If lastTraversalRunInfo.completeDateUtc is null: - STOP WITH REASON because traversal run did not report completion after maxPolls polls; include runNum and lastTraversalRunInfo in the reason. - traversalRunErrors = WdsGetTraversalRunErrors(jobName=jobName, runNum=runNum) - If traversalRunErrors.failedDownloadTasks is NOT empty: - For each downloadTask in traversalRunErrors.failedDownloadTasks: - For each failedDownloadTask in downloadTask.failedDownloadTasks: - status = WdsGetDownloadTaskStatus(taskId=failedDownloadTask.id) - If status.result is null: - Collect status.url, null, null - If status.result is not null: - Collect status.url, status.result.httpStatusCode, status.result.reasonPhrase - Output a table: | URL | HttpStatusCode | ReasonPhrase | - Proceed with successfully gathered data 12. Get all gathered data - cursor = null - traversalAllData = [] - REPEAT: - traversalData = WdsGetTraversalRunData(jobName=jobName, runNum=runNum, path=null, cursor=cursor, limit=100) - Append every item in traversalData.data to traversalAllData without dropping or overwriting previously fetched items - cursor = traversalData.dataCursor - UNTIL cursor is null 13. Analyze and answer - Analyze all retrieved and fetched data in traversalAllData. - Build a resume of the website with the following structure: - Main Topic: a brief description of the main topic of the website; - Contact Information: a list of contact information available on the website, including phone numbers, email addresses, and social media links; - Target Audience: a brief description of the target audience of the website, including their interests and needs. - FAQ: a list of frequently asked questions found on the website, with answers to each question; - Services: a list of services found on the website (if they are), with a full description of each service. - Products: a list of products found on the website (if they are), with a full description of each product; - SHOW the whole resume obtained to the user. ```` # assess Assess a web resource and propose a traversal config for data extraction. ## How to call In supported clients the command syntax can vary. If the WDS MCP server is registered as `wds`, use: | Client | Command | | --- | --- | | Visual Studio Code | `/mcp.wds.assess` | ## Arguments | Name | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | url | string | No | `http://playground` | Start URL | ## Returned Prompt Text The block below contains the complete prompt returned to the MCP client. Runtime placeholders map to arguments as follows: - `` — `url` ````nohighlight ROLE: Act as a web developer with experience in designing multi-page web resources, including paged data, templating, and expertise in pure CSS as well as in modern CSS frameworks: Bootstrap, Tailwind CSS, Foundation, Bulma, UIkit, Tachyons, Materialize, Metro, Spectre.css, Milligram, Pure.css, Skeleton, NES.css, W3.CSS, Picnic CSS, Basscss, Shoelace, etc. Your experience includes working with web resources that have complex structures, such as those with paged data, hierarchical menus, and leaf pages with data accessible from various navigation elements. You are skilled in identifying and extracting data fields from web pages using CSS selectors and XPath expressions. OBJECTIVE: Evaluate TARGET_URL with fetch-only inspection and propose a TraversalConfig for data extraction. RESTRICTIONS: - Only call WdsFetchChunked and WdsGetChunk. - Call MCP tools with named arguments matching the tool parameter names. - Use exact tool-returned values as inputs for later fetch steps. Do not rewrite or guess task IDs, URLs, or content spans. - Do not create selectors, values, or traversal branches from unseen content. Fetch the relevant page or representative branch page first. - Do not call job, traversal config, traversal run, or gathered-data tools from this prompt. - On any MCP tool error, STOP WITH REASON and surface the error response. GLOBAL CONSTANTS: TARGET_URL = '' EXAMPLE_TRAVERSAL_CONFIG: Main page fields: Name and Price from #products rows. Same-template pagination: #paging a. Detail branch: product links from #products tr a, with Description and Category on the detail page. ```json { "name": "/", "crawlParams": [ { "selector": "CSS: #paging a" } ], "scrapeParams": [ { "name": "Name", "selector": "CSS: #products tr td a" }, { "name": "Price", "selector": "CSS: #products tr td:nth-child(2)" } ], "branches": [ { "entryRule": { "selector": "CSS: #products tr a" }, "name": "product-details", "scrapeParams": [ { "name": "Description", "selector": "CSS: #product-details .desc" }, { "name": "Category", "selector": "CSS: #product-details .category" } ] } ] } ``` SELECTOR HEURISTICS (use in this order) 1. Unique IDs (#main, #products, #product-details) 2. Stable data-* or semantic classes (.product, [itemprop=name]) 3. Scoped structural selectors (#products tr td a) 4. nth-of-type / nth-child only as a last resort DEFINITIONS PopulateTraversalConfig(jobName, url, currentCfg) OBJECTIVE: Inspect fetched HTML and build the TraversalConfig needed for data extraction. PROCEDURE: 1. Fetch the HTML: - taskId = WdsFetchChunked(jobName=jobName, url=url) - REPEAT: - contentResult = WdsGetChunk(taskId=taskId, startIndex=0, length=1024) - If contentResult is not null: BREAK - Continue polling WdsGetChunk with the same arguments until contentResult is not null. - pageHtml = contentResult.contentSpan if it is not null; otherwise '' - contentLength = contentResult.contentLength if it is not null; otherwise length(pageHtml) - If more unseen content is required for the current analysis: - REPEAT: - startIndex = length(pageHtml) - If startIndex >= contentLength: BREAK - contentResult = WdsGetChunk(taskId=taskId, startIndex=startIndex, length=1024) - If contentResult is null: continue polling WdsGetChunk with the same arguments until contentResult is not null - Append contentResult.contentSpan to pageHtml - UNTIL enough relevant content has been inspected or startIndex >= contentLength - Analyze only the content present in pageHtml. If required evidence is in an unread span, read the next span before making conclusions. 2. Identify repeated record containers, individual data fields, same-template pagination links, and links to distinct detail templates. 3. Add field selectors to currentCfg.scrapeParams and same-template pagination selectors to currentCfg.crawlParams. 4. For each detail-template link selector: - Prefer the shortest direct path to data pages over menu/category paths that reach the same data. - Resolve one representative link against url, fetch it, create a subCfg with entryRule.selector set to the link selector, and populate subCfg from the fetched page. - Append subCfg to currentCfg.branches unless a branch with the same entryRule.selector already exists. - Decide same-level versus branch by page template and data role, not by URL string alone. 5. Return currentCfg. PROCEDURE (EXECUTE IN EXACT ORDER) 1. Derive proposed job name - Extract authority/host from TARGET_URL -> lowercase, normalized -> jobName - Keep jobName short and meaningful because it will be passed to the scrape prompt after the job is configured. 2. Build Traversal configuration - Create crawlCfg as a complete TraversalConfig with name = '/' - crawlCfg = PopulateTraversalConfig(jobName, TARGET_URL, crawlCfg) 3. Provide assessment result - Return the proposed jobName to pass to the scrape prompt after the job is configured as: Job name: - Return crawlCfg as JSON. - Summarize the proposed root fields, pagination selectors, and branch selectors. ```` # scrape Run a configured traversal job and return gathered data. ## How to call In supported clients the command syntax can vary. If the WDS MCP server is registered as `wds`, use: | Client | Command | | --- | --- | | Visual Studio Code | `/mcp.wds.scrape` | ## Arguments | Name | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | jobName | string | No | `playground` | Job name returned by assess | | outputFormat | string | No | `table` | Result output format. Allowed values: table, json, xml. | ## Returned Prompt Text The block below contains the complete prompt returned to the MCP client. Runtime placeholders map to arguments as follows: - `` — `jobName` - `` — `outputFormat` ````nohighlight ROLE: Act as a web developer with experience in designing multi-page web resources, including paged data, templating, and expertise in pure CSS as well as in modern CSS frameworks: Bootstrap, Tailwind CSS, Foundation, Bulma, UIkit, Tachyons, Materialize, Metro, Spectre.css, Milligram, Pure.css, Skeleton, NES.css, W3.CSS, Picnic CSS, Basscss, Shoelace, etc. Your experience includes working with web resources that have complex structures, such as those with paged data, hierarchical menus, and leaf pages with data accessible from various navigation elements. You are skilled in identifying and extracting data fields from web pages using CSS selectors and XPath expressions. OBJECTIVE: Run the prepared traversal job named JOB_NAME, retrieve all gathered data, and return it as OUTPUT_FORMAT. RESTRICTIONS: - The agent must use the exact values returned by MCP tools as inputs for later steps. Do not rewrite or guess job names, URLs, run numbers, cursors, or selectors. - MCP structured tool results are serialized as lower-camel JSON fields. Use returned field names such as runNum, completeDateUtc, failedDownloadTasks, data, and dataCursor when reading tool results. - The agent must perform all steps of the PROCEDURE in order, without skipping or parallel execution. - Call MCP tools with named arguments matching the tool parameter names. - WdsFetchChunked returns the download task ID as a string. It does not return raw page text. - Use WdsGetChunk(taskId=taskId, startIndex=startIndex, length=length) to read task content in chunks. Do not use shell/file tools to recover hidden or truncated content from MCP output. - Resolve relative links against the current page URL before passing them to WdsFetchChunked. - To change a JobConfig, read the current object with WdsGetJobConfig when it exists, create or modify the complete object in memory, then persist it with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg). Preserve all unrelated existing properties. - When setting nested JobConfig properties, create missing nested objects or arrays before setting their fields. - To change a TraversalConfig, create or modify the complete object in memory, then persist it with WdsUpsertTraversalConfig(jobName=jobName, traversalConfig=traversalConfig). - WdsRunTraversal and WdsTraversalAll return TraversalRunResult, not crawl data. Use runNum with WdsGetTraversalRunInfo, WdsGetTraversalRunErrors, and WdsGetTraversalRunData. - On any MCP tool error, the agent must STOP WITH REASON and surface the error response. - Do not create or modify jobs, job configs, or traversal configs from this prompt. - If the job or prepared traversal config does not exist, STOP WITH REASON and tell the user to configure the job with the TraversalConfig returned by assess first. - Do not stop reading gathered data while dataCursor is not null. GLOBAL CONSTANTS: JOB_NAME = '' OUTPUT_FORMAT = '' PROCEDURE (EXECUTE IN EXACT ORDER) 1. Find prepared job - jobs = WdsGetJobs() - Find job where job.jobName equals JOB_NAME (case-insensitive) - If not found: STOP WITH REASON because no prepared job exists for JOB_NAME. - jobName = job.jobName 2. Verify job configuration - jobCfg = WdsGetJobConfig(jobName=jobName) 3. Verify traversal configuration - crawlCfg = WdsGetTraversalConfig(jobName=jobName) - If crawlCfg is null: STOP WITH REASON because no prepared TraversalConfig exists for jobName. - If crawlCfg has no scrapeParams, crawlParams, and branches: STOP WITH REASON because the TraversalConfig is empty. 4. Start crawling - traversalRunResult = WdsRunTraversal(jobName=jobName) - runNum = traversalRunResult.runNum 5. Validate crawl completion - Wait for the traversal run to complete: - maxPolls = 60 - pollIntervalSec = 1 - pollCount = 0 - lastTraversalRunInfo = null - REPEAT while pollCount < maxPolls: - pollCount = pollCount + 1 - traversalRunInfo = WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) - lastTraversalRunInfo = traversalRunInfo - If traversalRunInfo.completeDateUtc is not null: - BREAK - Wait pollIntervalSec seconds before the next poll. - Continue polling with WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) until traversalRunInfo.completeDateUtc is not null or pollCount reaches maxPolls. - If lastTraversalRunInfo.completeDateUtc is null: - STOP WITH REASON because traversal run did not report completion after maxPolls polls; include runNum and lastTraversalRunInfo in the reason. - traversalRunErrors = WdsGetTraversalRunErrors(jobName=jobName, runNum=runNum) - If traversalRunErrors.failedDownloadTasks is NOT empty: - For each downloadTask in traversalRunErrors.failedDownloadTasks: - For each failedDownloadTask in downloadTask.failedDownloadTasks: - status = WdsGetDownloadTaskStatus(taskId=failedDownloadTask.id) - If status.result is null: - Collect status.url, null, null - If status.result is not null: - Collect status.url, status.result.httpStatusCode, status.result.reasonPhrase - Output a table: | URL | HttpStatusCode | ReasonPhrase | - Proceed with successfully gathered data 6. Retrieve all gathered data - cursor = null - traversalAllData = [] - REPEAT: - traversalData = WdsGetTraversalRunData(jobName=jobName, runNum=runNum, path=null, cursor=cursor, limit=100) - Append every item in traversalData.data to traversalAllData without dropping or overwriting previously fetched items - cursor = traversalData.dataCursor - UNTIL cursor is null 7. Provide results - If OUTPUT_FORMAT is 'table', return traversalAllData as a Markdown table. - If OUTPUT_FORMAT is 'json', return traversalAllData as JSON. - If OUTPUT_FORMAT is 'xml', return traversalAllData as XML. ```` # index Describes to an AI agent how to enroll a web resource into an index for further retrieval ## How to call In supported clients the command syntax can vary. If the WDS MCP server is registered as `wds`, use: | Client | Command | | --- | --- | | Visual Studio Code | `/mcp.wds.index` | ## Arguments | Name | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | url | string | No | `http://playground` | Web resource URL | ## Returned Prompt Text The block below contains the complete prompt returned to the MCP client. Runtime placeholders map to arguments as follows: - `` — `url` ````nohighlight ROLE: Act as a web developer with experience in designing multi-page web resources, including paged data, templating, and expertise in pure CSS as well as in modern CSS frameworks: Bootstrap, Tailwind CSS, Foundation, Bulma, UIkit, Tachyons, Materialize, Metro, Spectre.css, Milligram, Pure.css, Skeleton, NES.css, W3.CSS, Picnic CSS, Basscss, Shoelace, etc. Your experience includes working with web resources that have complex structures, such as those with paged data, hierarchical menus, and leaf pages with data accessible from various navigation elements. You are skilled in identifying and extracting data fields from web pages using CSS selectors and XPath expressions. OBJECTIVE: Create or reuse a crawl job for TARGET_URL, configure its content scope for retrieval indexing, start it once, and verify successful enrollment. RESTRICTIONS: - The agent must use the exact values returned by MCP tools as inputs for later steps. Do not rewrite or guess job names, URLs, run numbers, cursors, or selectors. - MCP structured tool results are serialized as lower-camel JSON fields. Use returned field names such as runNum, completeDateUtc, failedDownloadTasks, data, and dataCursor when reading tool results. - The agent must perform all steps of the PROCEDURE in order, without skipping or parallel execution. - Call MCP tools with named arguments matching the tool parameter names. - WdsFetchChunked returns the download task ID as a string. It does not return raw page text. - Use WdsGetChunk(taskId=taskId, startIndex=startIndex, length=length) to read task content in chunks. Do not use shell/file tools to recover hidden or truncated content from MCP output. - Resolve relative links against the current page URL before passing them to WdsFetchChunked. - To change a JobConfig, read the current object with WdsGetJobConfig when it exists, create or modify the complete object in memory, then persist it with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg). Preserve all unrelated existing properties. - When setting nested JobConfig properties, create missing nested objects or arrays before setting their fields. - To change a TraversalConfig, create or modify the complete object in memory, then persist it with WdsUpsertTraversalConfig(jobName=jobName, traversalConfig=traversalConfig). - WdsRunTraversal and WdsTraversalAll return TraversalRunResult, not crawl data. Use runNum with WdsGetTraversalRunInfo, WdsGetTraversalRunErrors, and WdsGetTraversalRunData. - On any MCP tool error, the agent must STOP WITH REASON and surface the error response. GLOBAL CONSTANTS: TARGET_URL = '' RESTART_MODE = 'Continue' DEFINITIONS: GetDataRootSelectors(jobName, url) Objective: Identify the core content (payload) areas of a web page and the navigation sections that lead to similar pages, then derive stable CSS selectors that represent those payload regions. Procedure: 1. Create a RESULT string 'CSS: '. 2. Fetch the HTML: - taskId = WdsFetchChunked(jobName=jobName, url=url) - REPEAT: - contentResult = WdsGetChunk(taskId=taskId, startIndex=0, length=1024) - If contentResult is not null: BREAK - Continue polling WdsGetChunk with the same arguments until contentResult is not null. - pageHtml = contentResult.contentSpan if it is not null; otherwise '' - contentLength = contentResult.contentLength if it is not null; otherwise length(pageHtml) - If more unseen content is required for the current analysis: - REPEAT: - startIndex = length(pageHtml) - If startIndex >= contentLength: BREAK - contentResult = WdsGetChunk(taskId=taskId, startIndex=startIndex, length=1024) - If contentResult is null: continue polling WdsGetChunk with the same arguments until contentResult is not null - Append contentResult.contentSpan to pageHtml - UNTIL enough relevant content has been inspected or startIndex >= contentLength - Analyze only the content present in pageHtml. If required evidence is in an unread span, read the next span before making conclusions. 3. Analyze the structure: - Parse pageHtml into its top-level sections: e.g.,
,