REST API
The WDS REST API exposes versioned endpoints for stored crawler jobs, task-level crawl and scrape processing, Traversal runs and schedules, retrieval-index search, and tenant cleanup. The base URL depends on the deployment option; a local Docker deployment normally exposes Dapi at http://localhost:2807.
All routes below use /api/v3. The core routes require Dapi’s Api feature, and schedule routes require Scheduling. Scheduling is available starting with the Business plan. See Plans for feature availability.
Swagger UI is available at /api/swagger. In debug builds, X-Tenant-Id can select a tenant; production uses the configured default tenant.
Jobs
| Operation |
Route |
Description |
| Get Jobs Info |
GET /jobs |
Lists jobs in the current tenant for discovery and lifecycle monitoring |
| Upsert a Job |
POST /jobs/{jobName}/config |
Creates or replaces the named job’s top-level crawler configuration, including start URLs, request settings, restart behavior, crawl scope, and retrieval indexing options |
| Get Config |
GET /jobs/{jobName}/config |
Reads the named job’s current crawler configuration before inspection or replacement |
| Fetch |
POST /jobs/{jobName}/fetch |
Downloads one absolute URL with the named job’s crawler settings and returns page content when ready, or a task content URL while pending |
| Start a Job |
POST /jobs/{jobName}/start |
Starts the named job from its configured start URLs. If the job was started before, applies its restart policy, then returns the initial download task IDs |
| Delete Job |
DELETE /jobs/{jobName} |
Deletes the named job and cleans up its Traversal config, schedule, tasks, run history, downloaded and scraped data, retrieval-index data, and service-side settings for the current tenant |
Tasks
| Operation |
Route |
Description |
| Crawl |
POST /tasks/{taskId}/crawl |
Extracts links from the downloaded page using CrawlParams, applies job scope and maxDepth rules, creates child download tasks, and completes the current task. If processing is pending, poll the crawl-result URL |
| Crawl Result |
GET /tasks/{taskId}/crawl-result |
Polls or resumes a previous Crawl request for this task using its stored CrawlParams until processing completes |
| Scrape |
POST /tasks/{taskId}/scrape |
Extracts named values from the downloaded page using ScrapeParams selectors and optional conversions, then completes the current task. If processing is pending, poll the scrape-result URL |
| Scrape Result |
GET /tasks/{taskId}/scrape-result |
Polls or resumes a previous Scrape request for this task using its stored ScrapeParams until processing completes |
| Content |
GET /tasks/{taskId}/content |
Reads all or part of the downloaded page for a task, typically one created by Fetch. If processing is pending, poll the same content URL |
| Download Status |
GET /tasks/{taskId}/download-status |
Inspects a task’s download state and HTTP attempt history to troubleshoot pending, failed, or completed downloads |
Retrieval
Retrieval routes require Dapi’s Retrieval feature. See Plans for feature availability.
| Operation |
Route |
Description |
| Tenant-wide Search |
POST /jobs/retrieve |
Searches retrieval-indexed crawled content across all jobs in the current tenant to find evidence relevant to a question |
| Job-scoped Search |
POST /jobs/{jobName}/retrieve |
Searches retrieval-indexed crawled content produced by one named job to find evidence relevant to a question |
Traversal
| Operation |
Route |
Description |
| Upsert Traversal Config |
POST /jobs/{jobName}/traversal/config |
Validates and stores the named job’s TraversalConfig multi-level crawl and scrape plan in the Traversal runner. This does not start a run |
| Get Traversal Config |
GET /jobs/{jobName}/traversal/config |
Reads the named job’s saved TraversalConfig so its multi-level crawl and scrape plan can be inspected or reused |
| Delete Traversal |
DELETE /jobs/{jobName}/traversal |
Deletes the named job’s saved TraversalConfig, schedule, tasks, and run history from the Traversal runner. This does not delete the job |
| Start Traversal Run |
POST /jobs/{jobName}/traversal/start |
Requests a Traversal run using the named job’s current job config and TraversalConfig, reusing an active run instead of starting a duplicate. Use the returned run number for later inspection |
| List Traversal Runs |
GET /jobs/{jobName}/traversal/runs |
Lists recent Traversal run summaries for the named job, newest first, so clients can find run numbers and completion state |
| Get Traversal Run Info |
GET /jobs/{jobName}/traversal/runs/{runNum}/info |
Inspects the progress and configuration used by one Traversal run |
| Get Traversal Run Errors |
GET /jobs/{jobName}/traversal/runs/{runNum}/errors |
Investigates failed downloads from one Traversal run and the parent pages that discovered them |
| Get Traversal Run Data |
GET /jobs/{jobName}/traversal/runs/{runNum}/data |
Reads a batch of scraped data produced by a Traversal run. Continue with the returned cursor until no cursor remains |
Traversal Scheduling
| Operation |
Route |
Description |
| Upsert Traversal Schedule |
POST /jobs/{jobName}/traversal/schedule |
Creates or replaces the named job’s TraversalSchedule for automatic runs. A saved TraversalConfig is required. This configures future runs and does not start one immediately |
| Get Traversal Schedule |
GET /jobs/{jobName}/traversal/schedule |
Reads the named job’s saved automatic Traversal schedule. A saved TraversalConfig is required |
| Delete Traversal Schedule |
DELETE /jobs/{jobName}/traversal/schedule |
Deletes the named job’s TraversalSchedule and stops future scheduled starts. A saved TraversalConfig is required. This does not delete the job, Traversal config, or previous runs |
Tenants
| Operation |
Route |
Description |
| Delete Tenant |
DELETE /tenants/{tenantId} |
Deletes a tenant and cleans up its jobs, download tasks, Traversal configs, schedules, tasks and run history, cached and scraped data, and retrieval-index data across backing services |
Typical Flow
- Upsert a job with its reusable
JobConfig.
- Start the job and collect its initial task IDs.
- Crawl, scrape, or read content from those tasks.
- When a task operation returns 202 (Accepted), retry the URL in
Location after Retry-After until the result is ready.
- For a multi-level plan, save a Traversal config, start a run, and page through its run data.