Release v3.1

Notes

WDS v3.1 makes collected pages easier to inspect and recurring crawls easier to manage. Browse downloaded content as HTML or Markdown, use job evaluation paths to find the entry pages, and bound completed-run history with Traversal retention. New Solidstack all-in-one images bundle MongoDB, with optional full-text search and local embeddings.

Upgrade the server services together: internal gRPC contracts changed. The REST API remains /api/v3. Review the polling-route and cache-storage changes below before upgrading. The MCP server now uses stateless Streamable HTTP.

MaxRuns limits run history rather than total storage; cached payloads and identifier records can remain. Delete Job can also leave download and cache state in this release.

Breaking Changes

API

• Crawl and scrape result polling moves from GET /tasks/{taskId}/crawl-result and GET /tasks/{taskId}/scrape-result to GET on /crawl and /scrape. Initial requests remain POST operations. Follow the Location header after 202 Accepted.
• A successful Fetch now explicitly returns raw text/html. Clients expecting a JSON-encoded string must read the response as text. The task content polling endpoint still returns a JSON ContentChunk.

MCP

• Streamable HTTP is stateless. Clients must not depend on a persistent server-side MCP session or a session ID across requests.

MSSQL

• Direct callers of the MSSQL-compatible REST API must poll with GET /api/mssql/v3/tasks/{taskId}/crawl or /scrape; the crawl-result and scrape-result routes are removed. The SQL CLR transport follows the returned Location automatically, so installed SQL function signatures are unchanged.

Added

API

• Job Content: GET /jobs/{jobName}/content/{path} reads existing download tasks as .html or .md, with links rewritten under the job content route. The path extension selects the format; reading can initiate or resume downloading for an existing task.
• JobInfo.EvaluationUrls supplies server-relative HTML and Markdown entry paths for configured start URLs that already have task records.

• Root TraversalConfig.MaxRuns accepts a positive retention limit. Before a new run starts, older completed runs and their Traversal tasks are pruned, with cleanup requested for download tasks unreferenced by other runs. Omitted or null keeps all runs.

• Solidstack AIO, AIO FTS, and AIO FTVS bundle MongoDB; FTS adds MongoDB Atlas Local search, and FTVS also includes Ollama with a preloaded embeddinggemma model. Regular AIO images target linux/amd64 and linux/arm64.
• Standard service images now have regular and -fips distributions. Regular images use the Docker Hardened Images runtime; FIPS images use the Red Hat UBI .NET runtime and target linux/amd64.

MCP

• WdsGetJobs returns the expanded JobInfo contract with EvaluationUrls for inspection through the REST content endpoint.
• WdsUpsertTraversalConfig accepts root-level MaxRuns; retention applies when a subsequent run starts, including scheduled runs.

MSSQL

No new installed SQL functions or UDTs.

Changed

API

• New S3 cache object keys include the job ID. Reads fall back to the v3.0 tenant-based layout when the new key is missing, so existing objects remain readable without migration.
• Idealer source hashing and Solidstack’s in-process hashing are configurable through SOURCE_HASH_MODE: FullLength, Sha512, or Raw. Sha512 stores fixed-size source hashes. The default remains FullLength. Changing modes on a populated database changes lookup keys and can allocate different IDs; no automatic migration is provided.
• Request tracing includes TraceId in ASP.NET request logs and preserves a supplied nonblank X-Trace-Id, generating one otherwise.
• Retention cleanup removes unshared scraped-data references and retrieval bundles while preserving data still referenced elsewhere. Storage cleanup has the limitations noted above.
• Download status distinguishes reset tasks with an existing download-result record (Created) from previously completed tasks whose result record was removed (Deleted). These states do not indicate whether cached page payloads remain.

MCP

• The server identifies itself as WDS MCP Server and reports the application assembly version.
• TraversalInfo.Config and child branches use TraversalConfigBase. Run snapshots contain the traversal plan; job-level MaxRuns is read from the saved root configuration.
• WdsGetDownloadTaskStatus reflects the reset-versus-deleted task-state distinction described above.

MSSQL

• Server-side task polling uses the updated routes while preserving the installed CLR function and UDT surface. Download-status fixes also apply to TaskStatus.

Removed

API

• The separate crawl-result and scrape-result routes. Use GET on /crawl and /scrape.

MCP

No tools or prompts removed.

MSSQL

• The MSSQL-compatible REST crawl-result and scrape-result polling routes. No installed SQL functions or UDTs removed.