Self-Hosted Web Crawler for Structured Data
and Searchable Content

Crawl public websites and internal web resources, extract the fields you need, and make collected content searchable for AI agents and applications.

Run Web Data Source in your own infrastructure. Connect through REST, MCP, or SQL Server.

Start in 5 minutes See Examples

From Website to Usable Data

Collect product details for a data pipeline, prepare readable content for an AI agent, or build a searchable index of your documentation.

Discover the Right Pages

Start from a URL and follow the links that matter. Choose navigation elements, limit URL depth, and control whether the crawler follows links to other hosts.

Extract the Fields You Need

Turn page content into named fields such as titles, prices, and descriptions using CSS or XPath selectors. Convert extracted content to Markdown or keep the main readable content for your next step.

Search Collected Content

Enable full-text or vector search over crawled content. Retrieve relevant passages with source URLs and capture times so your applications and AI agents can trace the evidence.

Built to Fit Your Workflow

Keep collection and retrieval in your infrastructure, connect your existing tools, and reuse your extraction plans.

Run in Your Infrastructure

Deploy with Docker or Kubernetes to collect public and internal web content. Run indexing and retrieval locally, with deployment options for offline environments.

Explore Deployment Options

Connect Your Existing Tools

Use the REST API from applications, connect AI agents through MCP, or crawl and extract website data directly in T-SQL with the SQL Server integration.

See Integration Examples

Reuse Your Extraction Plans

Save multi-level plans that navigate categories, listings, and detail pages. Run them again as needed, inspect collected data, and check errors when a page fails.

Explore Crawl and Extraction Plans

Need recurring collection? Automatic scheduling is available with Business plans and higher on supported deployments. Compare Plans

Use Website Data in Your Existing Tools

Ask an AI agent to find relevant content, extract fields through the API, or bring website data into a SQL query.

AI Agents: Search Website Content

With Retrieval enabled and your MCP client connected, ask your agent to index a documentation site and retrieve relevant passages. Example request:

Index https://example.com/docs.
Find the documented backup options.
Include the source URL for each passage.

WDS provides indexing and retrieval tools; your agent uses the returned content to answer.

Connect an AI Agent

REST API: Extract Structured Fields

For a downloaded page, send these extraction rules to POST /api/v3/tasks/{taskId}/scrape:

[
  { "name": "Title", "selector": "CSS: h1" },
  { "name": "Price", "selector": "CSS: .price span" }
]

Example response from a product page:

[
  { "name": "Title", "values": ["Example Product"] },
  { "name": "Price", "values": ["$49"] }
]

The quickstart covers downloading pages and waiting for pending results.

Try the API Quickstart

SQL Server: Query Website Data

With the SQL Server integration installed and @jobConfig configured, follow product links and extract titles and prices:

SELECT
    p.Task.Url URL,
    wds.ScrapeFirst(p.Task, 'CSS: h1', null) Name,
    wds.ScrapeFirst(p.Task, 'CSS: .price span', null) Price
FROM wds.Start(@jobConfig) r
    OUTER APPLY wds.Crawl(r.Task, 'CSS: table a', null) p
Explore the SQL Server Integration

Put Website Data to Work

Deploy WDS, crawl your first pages, and connect the results to your application, AI agent, or SQL Server workflow.

Start with Docker Docker Compose Kubernetes (Helm) Air-Gapped Connect an AI Agent

A free plan is available. Compare Features and Plans