Processing
Fetch
Downloads one absolute URL with the named job’s crawler settings and returns page content when ready, or a task content URL while pending
Endpoint
POST /api/v3/jobs/{jobName}/fetch
Path Parameters
| Name | Type | Description |
|---|---|---|
| jobName | string | Required. Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
Request Body
Required application/json string. Absolute HTTP or HTTPS URL to fetch as a single page under the named job. This does not start link crawling
Responses
200 (Ok)
Returns string containing the page content.
202 (Accepted)
Returns no body with a task content URL in Location. Retry after the number of seconds in Retry-After (currently 1) until the content is ready.
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
404 (Not Found)
Job not found
422 (Unprocessable Content)
There is an issue with processing the page content. Refer to the response text for more information
Crawl
Extracts links from the downloaded page using CrawlParams, applies job scope and maxDepth rules, creates child download tasks, and completes the current task. If processing is pending, poll the crawl-result URL
Endpoint
POST /api/v3/tasks/{taskId}/crawl
Path Parameters
| Name | Type | Description |
|---|---|---|
| taskId | string | Required. Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints |
Request Body
Required application/json body containing CrawlParams.
Responses
200 (OK)
Returns DownloadTask[] as an array of DownloadTask.
202 (Accepted)
Task processing is pending. Follow the Location header to the crawl-result or scrape-result endpoint and retry after the number of seconds in Retry-After (currently 1). Continue until the response is no longer 202 (Accepted).
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information.
403 (Forbidden)
Access restricted. Refer to the response text for more information.
404 (Not Found)
Task not found.
422 (Unprocessable Content)
There is an issue with processing the page content. Refer to the response text for more information.
Crawl Result
Polls or resumes a previous Crawl request for this task using its stored CrawlParams until processing completes
Endpoint
GET /api/v3/tasks/{taskId}/crawl-result
Path Parameters
| Name | Type | Description |
|---|---|---|
| taskId | string | Required. Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints |
Responses
200 (OK)
Returns DownloadTask[] as an array of DownloadTask.
202 (Accepted)
Task processing is pending. Follow the Location header to the crawl-result or scrape-result endpoint and retry after the number of seconds in Retry-After (currently 1). Continue until the response is no longer 202 (Accepted).
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information.
403 (Forbidden)
Access restricted. Refer to the response text for more information.
404 (Not Found)
Task not found.
422 (Unprocessable Content)
There is an issue with processing the page content. Refer to the response text for more information.
Scrape
Extracts named values from the downloaded page using ScrapeParams selectors and optional conversions, then completes the current task. If processing is pending, poll the scrape-result URL
Endpoint
POST /api/v3/tasks/{taskId}/scrape
Path Parameters
| Name | Type | Description |
|---|---|---|
| taskId | string | Required. Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints |
Request Body
Required application/json body containing an array of ScrapeParams.
Responses
200 (OK)
Returns TaskScrapeResult[]. Each item uses the public ScrapeResult shape.
202 (Accepted)
Task processing is pending. Follow the Location header to the crawl-result or scrape-result endpoint and retry after the number of seconds in Retry-After (currently 1). Continue until the response is no longer 202 (Accepted).
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information.
403 (Forbidden)
Access restricted. Refer to the response text for more information.
404 (Not Found)
Task not found.
422 (Unprocessable Content)
There is an issue with processing the page content. Refer to the response text for more information.
Selector Format
The selector argument is a selector of the following format: CSS|XPATH: selector. The first part defines the selector type, the second one should be a selector in the corresponding type.
Supported types:
Scrape Result
Polls or resumes a previous Scrape request for this task using its stored ScrapeParams until processing completes
Endpoint
GET /api/v3/tasks/{taskId}/scrape-result
Path Parameters
| Name | Type | Description |
|---|---|---|
| taskId | string | Required. Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints |
Responses
200 (OK)
Returns TaskScrapeResult[]. Each item uses the public ScrapeResult shape.
202 (Accepted)
Task processing is pending. Follow the Location header to the crawl-result or scrape-result endpoint and retry after the number of seconds in Retry-After (currently 1). Continue until the response is no longer 202 (Accepted).
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information.
403 (Forbidden)
Access restricted. Refer to the response text for more information.
404 (Not Found)
Task not found.
422 (Unprocessable Content)
There is an issue with processing the page content. Refer to the response text for more information.
Get Content
Reads all or part of the downloaded page for a task, typically one created by Fetch. If processing is pending, poll the same content URL
Endpoint
GET /api/v3/tasks/{taskId}/content
Path Parameters
| Name | Type | Description |
|---|---|---|
| taskId | string | Required. Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints |
Query Parameters
| Name | Type | Description |
|---|---|---|
| startIndex | int | Optional. Optional. Zero-based start index of the returned content substring. Omit or pass null to start at 0. A negative value or an index at or beyond the content length returns an empty ContentSpan |
| length | int | Optional. Optional. Maximum number of content characters to return from startIndex. Omit or pass null to return all remaining content; a value of 0 or less returns an empty ContentSpan |
Responses
200 (OK)
Returns ContentChunk as a ContentChunk.
202 (Accepted)
Task processing is pending. Retry the URL in the Location header after the number of seconds in Retry-After (currently 1) until the response is no longer 202 (Accepted).
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
404 (Not Found)
Task not found
422 (Unprocessable Content)
There is an issue with processing the page content. Refer to the response text for more information
Download Status
Inspects a task’s download state and HTTP attempt history to troubleshoot pending, failed, or completed downloads
Endpoint
GET /api/v3/tasks/{taskId}/download-status
Path Parameters
| Name | Type | Description |
|---|---|---|
| taskId | string | Required. Download task ID returned by job start, in download task records from crawl requests, or embedded in a 202 Location URL from async task endpoints |
Responses
200 (OK)
Returns DownloadTaskStatus as a DownloadTaskStatus.
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
404 (Not Found)
Task not found
Data Contracts
DownloadTask
Represents one page download task created by job start, single-URL fetch, or link crawling
Fields:
| Name | Type | Description |
|---|---|---|
| Id | string | Download task ID |
| Url | string | Download task URL |
CrawlParams
Defines how links are extracted from a page and which discovered URLs remain in scope
Fields:
| Name | Type | Description |
|---|---|---|
| Selector | string | Selector used to find link-bearing elements. Use ‘CSS: |
| AttributeName | string | Optional. Attribute read from each matched element to obtain a link. Use ‘val’ for inner text. Omitted, null, or empty uses href |
| MaxDepth | int | Optional. Maximum URL path depth for discovered links. For example, example.com and example.com/index.html have depth 0, while example.com/path/ has depth 1. Omitted or null applies no depth limit |
ScrapeParams
Defines one named field extracted from pages by selector, attribute, and optional conversion
Fields:
| Name | Type | Description |
|---|---|---|
| Name | string | Name of the output field populated from this extraction rule. Names must be unique within one scrape request or Traversal level |
| Selector | string | Selector used to read matching elements. Use ‘CSS: |
| AttributeName | string | Optional. Attribute read from each matched element. Use ‘val’, or omit/pass null or empty, to return inner text |
| Convert | string | Optional. Conversion applied to scraped values. Use ‘md()’ for Markdown or ‘sr()’ for main readable content using Mozilla Readability. Omitted or null applies no conversion |
ScrapeResult
Values extracted for one ScrapeParams entry
Fields:
| Name | Type | Description |
|---|---|---|
| Name | string | Optional. Field name copied from the matching ScrapeParams entry |
| Values | array of string | Extracted values returned by the selector, attribute, and optional conversion |
ContentChunk
Reports the full downloaded content length and the full or requested content span for one task
Fields:
| Name | Type | Description |
|---|---|---|
| ContentLength | int | Optional. Length of the full downloaded page content in characters |
| ContentSpan | string | Optional. Full downloaded content or the requested substring. An invalid or non-positive range produces an empty string |
DownloadTaskStatus
Current state and request history for a download task
Fields:
| Name | Type | Description |
|---|---|---|
| Url | string | URL associated with the download task |
| State | DownloadTaskState | Current download task state |
| Result | DownloadInfo | Optional. Final download attempt details, or null when no attempt result is available |
| IntermedResults | array of DownloadInfo | Optional. Earlier attempt details such as redirects or proxy retries. Null means no attempt history is available; an empty array means only the final result exists |
DownloadTaskState
Lifecycle states for a download task
Enumeration values:
| Name | Description |
|---|---|
| Handled | The downloader finished handling the task. When DownloadTaskStatus.Result is present, inspect IsSuccess and HttpStatusCode to determine request success |
| AccessDeniedForRobots | The target URL was blocked by robots.txt |
| AllRequestGatesExhausted | All request gateways, such as proxies or host IP addresses, were exhausted without a successful response |
| Created | The task was created but has not started |
| InProgress | The task is currently being downloaded |
| Deleted | The task was deleted |
DownloadInfo
HTTP request and response details for one download attempt
Fields:
| Name | Type | Description |
|---|---|---|
| Method | string | HTTP method |
| Url | string | Request URL |
| IsSuccess | bool | Whether the request completed successfully |
| HttpStatusCode | int | HTTP response status code |
| ReasonPhrase | string | HTTP reason phrase |
| RequestHeaders | array of HttpHeader | HTTP headers sent with the request |
| ResponseHeaders | array of HttpHeader | HTTP headers received in the response |
| RequestCookies | array of Cookie | Cookies sent with the request |
| ResponseCookies | array of Cookie | Cookies received in the response |
| RequestDateUtc | datetime | Date and time when the request was sent, in UTC |
| DownloadTimeSec | double | Download time in seconds |
| ViaProxy | bool | Whether the request was made through a proxy |
| WaitTimeSec | double | Total delay in seconds before the request was executed |
| CrawlDelaySec | int | Crawl-delay portion of the wait time, in seconds |
HttpHeader
Represents one HTTP header with its name and one or more values
Fields:
| Name | Type | Description |
|---|---|---|
| Name | string | HTTP header name, for example User-Agent or Authorization |
| Values | array of string | One or more values for this header |
Cookie
HTTP cookie sent with a request or received in a response
See HTTP cookies for protocol-level background.
Fields:
| Name | Type | Description |
|---|---|---|
| Name | string | Cookie name |
| Value | string | Optional. Cookie value, or null when absent |
| Domain | string | Optional. Cookie domain attribute, or null when absent |
| Path | string | Optional. Cookie path attribute, or null when absent |
| HttpOnly | bool | Whether the cookie has the HttpOnly attribute |
| Secure | bool | Whether the cookie has the Secure attribute |
| Expires | datetime | Optional. Cookie expiration date and time, or null for a session cookie or unknown expiration |