Crawl
Extracts links from the downloaded page using CrawlParams, applies job scope and maxDepth rules, creates child download tasks, and completes the current task. If processing is pending, poll the crawl-result URL
Endpoint
POST /api/v3/tasks/{taskId}/crawl
Path Parameters
| Name | Type | Description |
|---|---|---|
| taskId | string | Required. Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints |
Request Body
Required application/json body containing CrawlParams.
Responses
200 (OK)
Returns DownloadTask[] as an array of DownloadTask.
202 (Accepted)
Task processing is pending. Follow the Location header to the crawl-result or scrape-result endpoint and retry after the number of seconds in Retry-After (currently 1). Continue until the response is no longer 202 (Accepted).
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information.
403 (Forbidden)
Access restricted. Refer to the response text for more information.
404 (Not Found)
Task not found.
422 (Unprocessable Content)
There is an issue with processing the page content. Refer to the response text for more information.
Data Contracts
DownloadTask
Represents one page download task created by job start, single-URL fetch, or link crawling
Fields:
| Name | Type | Description |
|---|---|---|
| Id | string | Download task ID |
| Url | string | Download task URL |
CrawlParams
Defines how links are extracted from a page and which discovered URLs remain in scope
Fields:
| Name | Type | Description |
|---|---|---|
| Selector | string | Selector used to find link-bearing elements. Use ‘CSS: |
| AttributeName | string or null | Optional. Attribute read from each matched element to obtain a link. Use ‘val’ for inner text. Omitted, null, or empty uses href |
| MaxDepth | int or null | Optional. Maximum URL path depth for discovered links. For example, example.com and example.com/index.html have depth 0, while example.com/path/ has depth 1. Omitted or null applies no depth limit |