API
The WDS API exposes versioned endpoints for stored crawler jobs, task-level crawl and scrape processing, Traversal runs and schedules, retrieval-index search, and tenant cleanup. The base URL depends on the deployment option; a local Docker deployment normally exposes Dapi at http://localhost:2807.
All routes below use /api/v3. The core routes require the Api feature, and schedule routes require the Traversal Scheduling feature. Scheduling is available starting with the Business plan. See Plans for feature availability.
Swagger UI is available at /api/swagger. In debug builds, X-Tenant-Id can select a tenant; production uses the configured default tenant.
Jobs
| Operation |
Route |
Description |
| Get Jobs Info |
GET /jobs |
Lists jobs in the current tenant for discovery and lifecycle monitoring |
| Upsert a Job |
POST /jobs/{jobName}/config |
Creates or replaces the named job’s top-level crawler configuration, including start URLs, request settings, restart behavior, crawl scope, and retrieval indexing options |
| Get Config |
GET /jobs/{jobName}/config |
Reads the named job’s current crawler configuration before inspection or replacement |
| Start a Job |
POST /jobs/{jobName}/start |
Starts the named job from its configured start URLs. If the job was started before, applies its restart policy, then returns the initial download task IDs |
| Delete Job |
DELETE /jobs/{jobName} |
Deletes the named job and cleans up its Traversal config, schedule, tasks, run history, downloaded and scraped data, retrieval-index data, and service-side settings for the current tenant |
Processing
| Operation |
Route |
Description |
| Fetch |
POST /jobs/{jobName}/fetch |
Downloads one absolute URL with the named job’s crawler settings and returns page content when ready, or a task content URL while pending |
| Crawl |
POST /tasks/{taskId}/crawl |
Extracts links from the downloaded page using CrawlParams, applies job scope and maxDepth rules, creates child download tasks, and completes the current task. If processing is pending, poll the crawl-result URL |
| Crawl Result |
GET /tasks/{taskId}/crawl-result |
Polls or resumes a previous Crawl request for this task using its stored CrawlParams until processing completes |
| Scrape |
POST /tasks/{taskId}/scrape |
Extracts named values from the downloaded page using ScrapeParams selectors and optional conversions, then completes the current task. If processing is pending, poll the scrape-result URL |
| Scrape Result |
GET /tasks/{taskId}/scrape-result |
Polls or resumes a previous Scrape request for this task using its stored ScrapeParams until processing completes |
| Content |
GET /tasks/{taskId}/content |
Reads all or part of the downloaded page for a task, typically one created by Fetch. If processing is pending, poll the same content URL |
| Download Status |
GET /tasks/{taskId}/download-status |
Inspects a task’s download state and HTTP attempt history to troubleshoot pending, failed, or completed downloads |
Retrieval
Retrieval routes require the Retrieval feature. See Plans for feature availability.
| Operation |
Route |
Description |
| Tenant-wide Search |
POST /jobs/retrieve |
Searches retrieval-indexed crawled content across all jobs in the current tenant to find evidence relevant to a question |
| Job-scoped Search |
POST /jobs/{jobName}/retrieve |
Searches retrieval-indexed crawled content produced by one named job to find evidence relevant to a question |
Traversal
| Operation |
Route |
Description |
| Upsert Traversal Config |
POST /jobs/{jobName}/traversal/config |
Validates and stores the named job’s TraversalConfig multi-level crawl and scrape plan in the Traversal runner. This does not start a run |
| Get Traversal Config |
GET /jobs/{jobName}/traversal/config |
Reads the named job’s saved TraversalConfig so its multi-level crawl and scrape plan can be inspected or reused |
| Delete Traversal |
DELETE /jobs/{jobName}/traversal |
Deletes the named job’s saved TraversalConfig, schedule, tasks, and run history from the Traversal runner. This does not delete the job |
| Start Traversal Run |
POST /jobs/{jobName}/traversal/start |
Requests a Traversal run using the named job’s current job config and TraversalConfig, reusing an active run instead of starting a duplicate. Use the returned run number for later inspection |
| List Traversal Runs |
GET /jobs/{jobName}/traversal/runs |
Lists recent Traversal run summaries for the named job, newest first, so clients can find run numbers and completion state |
| Get Traversal Run Info |
GET /jobs/{jobName}/traversal/runs/{runNum}/info |
Inspects the progress and configuration used by one Traversal run |
| Get Traversal Run Errors |
GET /jobs/{jobName}/traversal/runs/{runNum}/errors |
Investigates failed downloads from one Traversal run and the parent pages that discovered them |
| Get Traversal Run Data |
GET /jobs/{jobName}/traversal/runs/{runNum}/data |
Reads a batch of scraped data produced by a Traversal run. Continue with the returned cursor until no cursor remains |
Traversal Scheduling
| Operation |
Route |
Description |
| Upsert Traversal Schedule |
POST /jobs/{jobName}/traversal/schedule |
Creates or replaces the named job’s TraversalSchedule for automatic runs. A saved TraversalConfig is required. This configures future runs and does not start one immediately |
| Get Traversal Schedule |
GET /jobs/{jobName}/traversal/schedule |
Reads the named job’s saved automatic Traversal schedule. A saved TraversalConfig is required |
| Delete Traversal Schedule |
DELETE /jobs/{jobName}/traversal/schedule |
Deletes the named job’s TraversalSchedule and stops future scheduled starts. A saved TraversalConfig is required. This does not delete the job, Traversal config, or previous runs |
Tenants
| Operation |
Route |
Description |
| Delete Tenant |
DELETE /tenants/{tenantId} |
Deletes a tenant and cleans up its jobs, download tasks, Traversal configs, schedules, tasks and run history, cached and scraped data, and retrieval-index data across backing services |
Typical Flow
- Upsert a job with its reusable
JobConfig.
- Start the job and collect its initial task IDs.
- Crawl, scrape, or read content from those tasks.
- When a task operation returns 202 (Accepted), retry the URL in
Location after Retry-After until the result is ready.
- For a multi-level plan, save a Traversal config, start a run, and page through its run data.