Traversal
Upsert Traversal Config
Validates and stores the named job’s TraversalConfig multi-level crawl and scrape plan in the Traversal runner. This does not start a run
Endpoint
POST /api/v3/jobs/{jobName}/traversal/config
Path Parameters
| Name | Type | Description |
|---|---|---|
| jobName | string | Required. Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
Request Body
Required application/json body containing a TraversalConfig.
Responses
204 (No Content)
Returns void with no response body.
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
Get Traversal Config
Reads the named job’s saved TraversalConfig so its multi-level crawl and scrape plan can be inspected or reused
Endpoint
GET /api/v3/jobs/{jobName}/traversal/config
Path Parameters
| Name | Type | Description |
|---|---|---|
| jobName | string | Required. Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
Responses
200 (Ok)
Returns TraversalConfig? as a TraversalConfig, or null when no configuration is saved.
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
Delete Traversal
Deletes the named job’s saved TraversalConfig, schedule, tasks, and run history from the Traversal runner. This does not delete the job
Endpoint
DELETE /api/v3/jobs/{jobName}/traversal
Path Parameters
| Name | Type | Description |
|---|---|---|
| jobName | string | Required. Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
Responses
204 (No Content)
Returns void with no response body.
403 (Forbidden)
Access restricted. Refer to the response text for more information
Start Traversal Run
Requests a Traversal run using the named job’s current job config and TraversalConfig, reusing an active run instead of starting a duplicate. Use the returned run number for later inspection
Endpoint
POST /api/v3/jobs/{jobName}/traversal/start
Path Parameters
| Name | Type | Description |
|---|---|---|
| jobName | string | Required. Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
Responses
200 (Ok)
Returns TraversalRunResult as a TraversalRunResult.
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
List Traversal Runs
Lists recent Traversal run summaries for the named job, newest first, so clients can find run numbers and completion state
Endpoint
GET /api/v3/jobs/{jobName}/traversal/runs
Path Parameters
| Name | Type | Description |
|---|---|---|
| jobName | string | Required. Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
Query Parameters
| Name | Type | Description |
|---|---|---|
| limit | int | Optional. Default: 10. Optional. Number of newest runs to return, ordered by run number descending. Default: 10 |
Responses
200 (Ok)
Returns TraversalInfo[] as an array of TraversalInfo.
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
Get Traversal Run Info
Inspects the progress and configuration used by one Traversal run
Endpoint
GET /api/v3/jobs/{jobName}/traversal/runs/{runNum}/info
Path Parameters
| Name | Type | Description |
|---|---|---|
| jobName | string | Required. Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
| runNum | unsigned long | Required. Traversal run number used to read run status, errors, and data for the same job |
Responses
200 (Ok)
Returns TraversalInfo as a TraversalInfo.
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
Get Traversal Run Errors
Investigates failed downloads from one Traversal run and the parent pages that discovered them
Endpoint
GET /api/v3/jobs/{jobName}/traversal/runs/{runNum}/errors
Path Parameters
| Name | Type | Description |
|---|---|---|
| jobName | string | Required. Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
| runNum | unsigned long | Required. Traversal run number used to read run status, errors, and data for the same job |
Responses
200 (Ok)
Returns TraversalErrors as TraversalErrors.
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
Get Traversal Run Data
Reads a batch of scraped data produced by a Traversal run. Continue with the returned cursor until no cursor remains
Endpoint
GET /api/v3/jobs/{jobName}/traversal/runs/{runNum}/data
Path Parameters
| Name | Type | Description |
|---|---|---|
| jobName | string | Required. Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data |
| runNum | unsigned long | Required. Traversal run number used to read run status, errors, and data for the same job |
Query Parameters
| Name | Type | Description |
|---|---|---|
| path | string | Optional. Optional. Path to a TraversalConfig level in the Traversal tree. Use ‘/’ for the root level. Nested paths must start and end with ‘/’, for example ‘/products/’. Omit to read full leaf-level data for the run. Set a level path to stop at that Traversal level and limit the shape or size of returned JSON objects. See TraversalConfig for level names |
| cursor | string | Optional. Optional. Opaque DataCursor returned by the previous response. Omit or pass null to read the first batch |
| limit | int | Optional. Default: 10. Optional. Number of Traversal tasks at the selected path to read for this batch. Default: 10. One task can produce multiple JSON documents, so Data length can be greater than this limit |
Responses
200 (Ok)
Returns TraversalData as TraversalData. Continue with DataCursor until the returned cursor is null.
400 (Bad Request)
Invalid request parameters. Refer to the response text for more information
403 (Forbidden)
Access restricted. Refer to the response text for more information
Data Contracts
TraversalConfig
Multi-level crawl and scrape plan that defines level names, link selectors, field extraction rules, and nested branches
Fields:
| Name | Type | Description |
|---|---|---|
| Name | string | Name of this Traversal level. Use ‘/’ for the root; use names ending in ‘/’, such as ‘products/’, for branch levels so their data paths are addressable |
| CrawlParams | array of CrawlParams | Optional. Link extraction rules used to discover more pages at this level. Omitted, null, or empty performs no same-level link discovery |
| ScrapeParams | array of ScrapeParams | Optional. Field extraction rules used to collect data at this level. Omitted, null, or empty extracts no fields; names must be unique within the level |
| Branches | array of TraversalBranch | Optional. Nested Traversal branches reachable from this level. Omitted, null, or empty defines no child levels |
CrawlParams
Defines how links are extracted from a page and which discovered URLs remain in scope
Fields:
| Name | Type | Description |
|---|---|---|
| Selector | string | Selector used to find link-bearing elements. Use ‘CSS: |
| AttributeName | string | Optional. Attribute read from each matched element to obtain a link. Use ‘val’ for inner text. Omitted, null, or empty uses href |
| MaxDepth | int | Optional. Maximum URL path depth for discovered links. For example, example.com and example.com/index.html have depth 0, while example.com/path/ has depth 1. Omitted or null applies no depth limit |
ScrapeParams
Defines one named field extracted from pages by selector, attribute, and optional conversion
Fields:
| Name | Type | Description |
|---|---|---|
| Name | string | Name of the output field populated from this extraction rule. Names must be unique within one scrape request or Traversal level |
| Selector | string | Selector used to read matching elements. Use ‘CSS: |
| AttributeName | string | Optional. Attribute read from each matched element. Use ‘val’, or omit/pass null or empty, to return inner text |
| Convert | string | Optional. Conversion applied to scraped values. Use ‘md()’ for Markdown or ‘sr()’ for main readable content using Mozilla Readability. Omitted or null applies no conversion |
TraversalBranch
Describes a child Traversal branch and the link extraction rule used to reach it from the current level
TraversalBranch inherits the fields of TraversalConfig and adds this field:
| Name | Type | Description |
|---|---|---|
| EntryRule | CrawlParams | Link extraction rule that discovers pages belonging to this child branch |
TraversalRunResult
Result returned when starting or attaching to a Traversal run
Fields:
| Name | Type | Description |
|---|---|---|
| RunNum | long | Traversal run number |
| IsNew | bool | True when a new run was started; false when an existing in-progress run was returned |
TraversalInfo
Status, counters, and config snapshot for one Traversal run
Fields:
| Name | Type | Description |
|---|---|---|
| RunNum | long | Traversal run number |
| StartDateUtc | datetime | Date and time when the run started, in UTC |
| CompleteDateUtc | datetime | Optional. Date and time when the run completed, in UTC; null means the run is still in progress |
| InProcessCrawlTasksCount | long | Number of download tasks currently in progress |
| SuccessfulDownloadTaskCount | long | Number of download tasks that completed successfully |
| FailedDownloadTaskCount | long | Number of failed download tasks |
| Config | TraversalConfig | TraversalConfig snapshot used for this run |
TraversalErrors
Failed download tasks from a Traversal run, grouped by parent page
Fields:
| Name | Type | Description |
|---|---|---|
| FailedDownloadTasks | array of FailedDownloadTask | Parent pages and the child download tasks that failed while crawling them |
FailedDownloadTask
Failed child download tasks discovered from one parent page
Fields:
| Name | Type | Description |
|---|---|---|
| ParentDownloadTaskUrl | string | URL of the parent page where the failed links were discovered, or the literal ‘root’ for failed start tasks |
| FailedDownloadTasks | array of DownloadTask | Child download tasks that failed |
DownloadTask
Represents one page download task created by job start, single-URL fetch, or link crawling
Fields:
| Name | Type | Description |
|---|---|---|
| Id | string | Download task ID |
| Url | string | Download task URL |
TraversalData
Paged scraped data returned for a Traversal run
Fields:
| Name | Type | Description |
|---|---|---|
| Data | array of string | Scraped data records serialized as JSON strings |
| DataCursor | string | Optional. Opaque cursor for fetching the next batch of data; null means there is no next page |