API
The WDS API exposes versioned endpoints for stored crawler jobs, task-level crawl and scrape processing, Traversal runs and schedules, retrieval-index search, and tenant cleanup. The base URL depends on the deployment option; a local Docker deployment normally exposes Dapi at http://localhost:2807.
All routes below use /api/v3. The core routes require the Api feature, and schedule routes require the Traversal Scheduling feature. Scheduling is available starting with the Business plan. See Plans for feature availability.
Swagger UI is available at /api/swagger. In debug builds, X-Tenant-Id can select a tenant; production uses the configured default tenant.
Requests can supply X-Trace-Id; WDS preserves a nonblank value or generates one when absent. The response header and ASP.NET request logs use the same trace ID for troubleshooting.
Jobs
| Operation |
Route |
Description |
| Get Jobs Info |
GET /jobs |
Lists jobs in the current tenant for discovery and lifecycle monitoring |
| Upsert a Job |
POST /jobs/{jobName}/config |
Creates or replaces the named job’s top-level crawler configuration, including start URLs, request settings, restart behavior, crawl scope, and retrieval indexing options |
| Get Config |
GET /jobs/{jobName}/config |
Reads the named job’s current crawler configuration before inspection or replacement |
| Job Content |
GET /jobs/{jobName}/content/{path} |
Reads page content for an existing download task in the named job, using .html for HTML or .md for Markdown, with links rewritten under the job content route. The root .html and .md links are available in each job’s EvaluationUrls returned by Get Jobs Info. All configured start URLs must share a scheme and authority. The request can initiate or resume downloading for that task. If content is pending, the request returns an error rather than a polling location |
| Start a Job |
POST /jobs/{jobName}/start |
Starts the named job from its configured start URLs. If the job was started before, applies its restart policy, then returns the initial download task IDs |
| Delete Job |
DELETE /jobs/{jobName} |
Deletes the named job, its Traversal configuration, schedules and run history, and requests downstream data cleanup. In this release, download-task records and cached payloads can remain; this is not a complete storage-erasure operation |
Processing
| Operation |
Route |
Description |
| Fetch |
POST /jobs/{jobName}/fetch |
Downloads one absolute URL with the named job settings. If pending, follow the task content URL in Location and poll with GET |
| Crawl |
POST /tasks/{taskId}/crawl |
Extracts links from the downloaded page using CrawlParams, applies job scope and maxDepth rules, creates child download tasks, and completes the current task. If pending, poll with GET on the same /crawl path after Retry-After |
| Crawl Result |
GET /tasks/{taskId}/crawl |
Polls or resumes a previous POST Crawl request using its stored CrawlParams. Use GET on the same /crawl path until processing completes |
| Scrape |
POST /tasks/{taskId}/scrape |
Extracts named values from the downloaded page using ScrapeParams selectors and optional conversions, then completes the current task. If pending, poll with GET on the same /scrape path after Retry-After |
| Scrape Result |
GET /tasks/{taskId}/scrape |
Polls or resumes a previous POST Scrape request using its stored ScrapeParams. Use GET on the same /scrape path until processing completes |
| Content |
GET /tasks/{taskId}/content |
Reads all or part of the downloaded page for a task, typically one created by Fetch. If processing is pending, poll the same content URL |
| Download Status |
GET /tasks/{taskId}/download-status |
Inspects a task’s download state and HTTP attempt history to troubleshoot pending, failed, or completed downloads |
Retrieval
Retrieval routes require the Retrieval feature. See Plans for feature availability.
| Operation |
Route |
Description |
| Tenant-wide Search |
POST /jobs/retrieve |
Searches retrieval-indexed crawled content across all jobs in the current tenant to find evidence relevant to a question |
| Job-scoped Search |
POST /jobs/{jobName}/retrieve |
Searches retrieval-indexed crawled content produced by one named job to find evidence relevant to a question |
Traversal
| Operation |
Route |
Description |
| Upsert Traversal Config |
POST /jobs/{jobName}/traversal/config |
Validates and stores the named job’s TraversalConfig multi-level crawl and scrape plan in the Traversal runner. This does not start a run |
| Get Traversal Config |
GET /jobs/{jobName}/traversal/config |
Reads the named job’s saved TraversalConfig so its multi-level crawl and scrape plan can be inspected or reused |
| Delete Traversal |
DELETE /jobs/{jobName}/traversal |
Deletes the named job’s saved TraversalConfig, schedule, tasks, and run history from the Traversal runner. This does not delete the job |
| Start Traversal Run |
POST /jobs/{jobName}/traversal/start |
Requests a Traversal run using the saved job and Traversal configuration, reusing an active run instead of starting a duplicate. A new run can prune older completed runs when MaxRuns is set; export needed results first. Use the returned run number for later inspection |
| List Traversal Runs |
GET /jobs/{jobName}/traversal/runs |
Lists recent Traversal run summaries for the named job, newest first, so clients can find run numbers and completion state |
| Get Traversal Run Info |
GET /jobs/{jobName}/traversal/runs/{runNum}/info |
Inspects the progress and configuration used by one Traversal run |
| Get Traversal Run Errors |
GET /jobs/{jobName}/traversal/runs/{runNum}/errors |
Investigates failed downloads from one Traversal run and the parent pages that discovered them |
| Get Traversal Run Data |
GET /jobs/{jobName}/traversal/runs/{runNum}/data |
Reads a batch of scraped data produced by a Traversal run. Continue with the returned cursor until no cursor remains |
Traversal Scheduling
| Operation |
Route |
Description |
| Upsert Traversal Schedule |
POST /jobs/{jobName}/traversal/schedule |
Creates or replaces the named job’s TraversalSchedule for automatic runs. A saved TraversalConfig is required. This configures future runs and does not start one immediately |
| Get Traversal Schedule |
GET /jobs/{jobName}/traversal/schedule |
Reads the named job’s saved automatic Traversal schedule. A saved TraversalConfig is required |
| Delete Traversal Schedule |
DELETE /jobs/{jobName}/traversal/schedule |
Deletes the named job’s TraversalSchedule and stops future scheduled starts. A saved TraversalConfig is required. This does not delete the job, Traversal config, or previous runs |
Tenants
| Operation |
Route |
Description |
| Delete Tenant |
DELETE /tenants/{tenantId} |
Deletes a tenant and cleans up its jobs, download tasks, Traversal configs, schedules, tasks and run history, cached and scraped data, and retrieval-index data across backing services |
Typical Flow
- Upsert a job with its reusable
JobConfig.
- Start the job and collect its initial task IDs.
- Crawl, scrape, or read content from those tasks.
- When a task operation returns 202 (Accepted), retry the URL in
Location after Retry-After until the result is ready.
- For a multi-level plan, save a Traversal config, start a run, and page through its run data.