Release v2.1

Notes

We’re excited to unveil Retrieve, a powerful new capability that seamlessly bridges web crawling and indexing. With Retrieve, every crawled resource can now be automatically added to an index, unlocking instant compatibility with Retrieval-Augmented Generation (RAG) workflows.

This means you can now crawl, index, and query your data - all within WDS! Whether you need to search across a single domain or your entire crawled dataset, WDS has you covered. You can access it effortlessly via REST API or through MCP Tools bringing unified data navigation, intelligent search, and deep analysis into one integrated ecosystem.

Together, these features make Web Data Source (WDS) a complete solution for exploring, searching, and leveraging your organization’s knowledge with both public and private LLMs.

For technical details, explore the documentation for the components below to learn how everything fits together.

Added

• Retrieval
• New APIs
  • Get jobs info
  • Get job config
  • Fetch page content
• New parameters
  • Convert parameter has been added for all scrape-based functions
• New MCP tools
  • UpsertJobConfig
  • StartExistingJob
  • CrawlMdrConfigSetMaxDepth
  • JobFetch
  • CrawlAllMdr
  • GetJobsInfo
  • GetJobConfig
  • JobConfigRetrievalConfigSetEnrollInIndex
  • JobConfigRetrievalConfigSetMaxTokensPerChunk
  • JobConfigRetrievalConfigUpsertContentScope
  • JobConfigRetrievalConfigSetWaitForEnrollment
  • JobConfigRetrievalConfigSetForce
• New MCP propmpts
  • Sliced Resume
  • Index
  • Reindex
  • Query

Modified

• Helm Chart:
  • Allow providing connection strings and license key from K8S secrets
• WDS MCP tools registration
  • All WDS MCP tools are now registering with the Wds prefix in their names

Removed

Nothing

Known Issues