Release v3.0

Notes

WDS v3.0 turns reusable job configuration and multi-level Traversal into first-class workflows. Store a JobConfig, run a Traversal plan when needed, or schedule it so fresh, structured data is ready for the next workflow. REST and MCP now share the same configuration-first model, while chunked fetching and job- or tenant-scoped retrieval give agents clearer paths from a URL to usable content and search results.

Breaking Changes

WDS v3.0 introduces Traversal Runs—a powerful new way to capture repeatable snapshots of web resources and transform them into structured, ready-to-use data. Replacing the previous CrawlMdr workflow, Traversal lets you configure a job once, launch tracked runs whenever fresh data is needed, and inspect status, errors, and cursor-paginated results. Runs can also be scheduled for recurring collection, helping databases, AI agents, and automated workflows stay continuously supplied with current web data through REST or MCP.

IMPORTANT! WDS v3 job schemas are not compatible with v2 job schemas. Clients must migrate existing v2 job configurations to the documented v3 schemas before using them with WDS v3.

API

• REST API routes now use v3. Clients must move from the v2 routes to the documented v3 routes.
• Get Jobs Info moved from /jobs/info to /jobs, and Fetch moved from a GET query to a JSON POST request.
• Start a Job no longer accepts a JobConfig request body. Persist the configuration with Upsert a Job, then start it by job name.
• Crawl and scrape calls are now POST operations. The old GET /tasks/{taskId}/crawl, GET /tasks/{taskId}/scrape, and both /scrape-multiple routes are replaced by Crawl and Scrape, which use current request contracts.
• Retrieval moved from the separate v2 query routes to JSON POST requests at tenant-wide and job-scoped /jobs/.../retrieve routes.
• GET /tasks/{taskId}/info is replaced by Download Status, which returns the current task-status shape.
• DownloadTaskStatus now uses State, Result, and IntermedResults; REST clients must no longer depend on the old TaskState or public Error fields.
• Delete Tenant now returns 204 No Content instead of 200 OK; clients must not expect a response body.

MCP

• The legacy WdsStartJob and WdsStartExistingJob tools are no longer registered. Use WdsUpsertJobConfig followed by the applicable job or Traversal tool.
• WdsRunTraversal now takes a stored Traversal configuration selected by jobName; it no longer accepts direct task and Traversal configuration arguments.
• WdsGetTraversalRunData uses the current run-data cursor arguments, and WdsRetrieve no longer accepts nextCursor.
• The old fetch workflow is replaced by WdsFetchChunked followed by WdsGetChunk.
• WdsGetJobsInfo is now WdsGetJobs; clients must use the new registered tool name.

MSSQL

No documented public changes.

Added

API

• Configuration-first jobs: Upsert a Job and Delete Job.
• A complete Traversal API: save, read, and delete configuration; start runs; then list runs and read their info, errors, and cursor-paginated data. See Traversal.
• Traversal scheduling to save, read, and delete recurring runs. Scheduling is available with the Business plan.
• Task-result endpoints for content chunks, crawl results, scrape results, and download status, plus JSON-body Scrape requests using ScrapeParams[].

MCP

• WdsFetchChunked and WdsGetChunk for predictable, chunked page retrieval.
• Traversal configuration, execution, inspection, and scheduling tools, including WdsTraversalAll for an end-to-end run.
• WdsDeleteJob and WdsDeleteTenant lifecycle-management tools.
• The assess and scrape prompts for guided content-assessment and extraction workflows.

MSSQL

• Installed HostConfig, RetrievalConfig, and RetrievalContentScope UDTs, exposed through the corresponding optional JobConfig fields.

Changed

API

• Job configuration now includes the current retrieval enrollment, host lifecycle, restart, error-handling, cross-domain, cookie, proxy, and crawler-protection contracts.
• Retrieval responses use the centralized RetrievalItem contract, and task processing separates an accepted request from its later result endpoint.

MCP

• WdsTraversalAll accepts a nullable int maxDepth, and WdsGetDownloadTaskStatus returns the current task-status contract.
• The existing index, reindex, query, resume, and sliced-resume prompts now use configuration-first Traversal and chunked-fetch workflows. Shared prompt procedures read 1,024-character chunks and wait one second between run-status polls.

MSSQL

• DownloadTaskStatus now includes the downloaded Url; its state and attempt-history fields are documented from the current CLR contract.

Removed

API

• The v2 job-info, task-info, GET crawl/scrape, scrape-multiple, and separate retrieval-query routes are no longer documented. Use the replacement v3 endpoints linked above.
• FetchResult and FetchContentResult are replaced by ContentChunk.

MCP

• WdsCrawl, WdsScrape, WdsCrawlMdr, WdsCrawlMdrConfig, WdsCrawlAllMdr, WdsGetCrawlMdrData, and the WdsJobConfigRetrievalConfig* helpers are no longer registered; use the current Traversal, fetch, and job-configuration tools.
• WdsJobFetch, WdsStartJobForFastWebResourceEvaluation, and WdsUpdateJobToContinueWebResourceEvaluation are no longer registered; use WdsFetchChunked, WdsTraversalAll, and WdsUpsertJobConfig, respectively.
• The former scrape-data prompt is replaced by the current scrape prompt.

MSSQL

No documented public changes.