Quickstart — up and running in a couple of minutes

Deploy WDS and run your first crawl/scrape entirely in Swagger UI.

Prerequisites

• Docker installed and running
• WDS deployed following the guide: Deploying WDS API Server in Docker Compose using the reconciled MINI (Free) option, with its MongoDB connection and embedding-service URL placeholders replaced

Once deployed, the API is available at: http://localhost:2807

Step 1 — Open Swagger UI

Open: http://localhost:2807/api/swagger

You’ll use six endpoints:

• Jobs -> POST {jobName}/config — create or update the job configuration
• Jobs -> POST {jobName}/start — start the stored job configuration and get the initial task IDs
• Tasks -> POST {taskId}/crawl — discover follow-up pages (links) from a page
• Tasks -> GET {taskId}/crawl-result — poll for pending crawl results
• Tasks -> POST {taskId}/scrape — extract data from a page
• Tasks -> GET {taskId}/scrape-result — poll for pending scrape results

Step 2 — Create a job configuration

In Swagger UI:

1. Expand Jobs -> POST {jobName}/config, then click “Try it out”.
2. Path parameter jobName: enter playground (or any unique name).
3. Request body:

   {
    "startUrls": ["http://playground"],
    "type": "Intranet"
   }
   

4. Click “Execute”.

Response: 202 Accepted confirms the job configuration was saved.

Step 3 — Start the job

In Swagger UI:

1. Expand Jobs -> POST {jobName}/start, then click “Try it out”.
2. Path parameter jobName: enter the same job name from Step 2, for example playground.
3. Click “Execute”.

Response: 200 OK returns an array of download task IDs. Copy one value; this is your first page task ID.

Step 4 — Discover pages (Crawl)

In Swagger UI:

1. Expand Tasks -> POST {taskId}/crawl, then click “Try it out”.
2. Path parameter taskId: paste the task ID from the Start response.
3. Request body:

   {
    "selector": "css: a[href*='/cloak_of_the_phantom.html']",
    "attributeName": "href"
   }
   

4. Click “Execute”.

If the response is 202 Accepted, wait for the Retry-After interval and call the URL in the Location header (Tasks -> GET {taskId}/crawl-result). Repeat until it returns 200 OK.

The 200 OK response contains an array of new DownloadTask items (in this example, a single item). Copy its id value for the scraping step.

Step 5 — Extract content (Scrape)

In Swagger UI:

1. Expand Tasks -> POST {taskId}/scrape, then click “Try it out”.
2. Path parameter taskId: paste the selected task id from Step 4.
3. Request body:

   [
    { "name": "Title", "selector": "css: h1" },
    { "name": "Price", "selector": "css: div.price span" },
    { "name": "Description", "selector": "css: div.desc p" }
   ]
   

4. Click “Execute”.

If the response is 202 Accepted, wait for the Retry-After interval and call the URL in the Location header (Tasks -> GET {taskId}/scrape-result). Repeat until it returns 200 OK.

The 200 OK response contains an array of objects with values for each field. For example:

[
  {
    "name": "Title",
    "values": [ "Cloak of the Phantom" ]
  },
  {
    "name": "Price",
    "values": [ "100 Fairy Coins" ]
  },
  {
    "name": "Description",
    "values": [ "Made from the feathers of a phoenix, it grants the power of rebirth." ]
  }
]

You’ve successfully extracted data — all within Swagger UI.

Conclusion

That’s it — deploy, configure, start, crawl, and scrape using only Swagger UI. For more, see the full API docs and Services.