WdsUpsertTraversalConfig

Validates and stores the named job’s TraversalConfig multi-level crawl and scrape plan in the Traversal runner. This does not start a run

Arguments

Name Type Required Default Description
jobName string Yes None Unique job name within the tenant. Reuse the same value to manage job config, start crawls, fetch URLs, inspect tasks and Traversal runs, and query indexed data
traversalConfig TraversalConfig Yes None Multi-level crawl and scrape plan that defines level names, link selectors, field extraction rules, and nested branches

Return Type

Returns void.

Technical return shape: void.

Data Contracts

TraversalConfig

Multi-level crawl and scrape plan that defines level names, link selectors, field extraction rules, and nested branches

Fields:

Name Type Description
Name string Name of this Traversal level. Use ‘/’ for the root; use names ending in ‘/’, such as ‘products/’, for branch levels so their data paths are addressable
CrawlParams array of CrawlParams or null Optional. Link extraction rules used to discover more pages at this level. Omitted, null, or empty performs no same-level link discovery
ScrapeParams array of ScrapeParams or null Optional. Field extraction rules used to collect data at this level. Omitted, null, or empty extracts no fields; names must be unique within the level
Branches array of TraversalBranch or null Optional. Nested Traversal branches reachable from this level. Omitted, null, or empty defines no child levels

CrawlParams

Defines how links are extracted from a page and which discovered URLs remain in scope

Fields:

Name Type Description
Selector string Selector used to find link-bearing elements. Use ‘CSS: ', 'XPATH: ', or '*' to select all anchor elements
AttributeName string or null Optional. Attribute read from each matched element to obtain a link. Use ‘val’ for inner text. Omitted, null, or empty uses href
MaxDepth int or null Optional. Maximum URL path depth for discovered links. For example, example.com and example.com/index.html have depth 0, while example.com/path/ has depth 1. Omitted or null applies no depth limit

ScrapeParams

Defines one named field extracted from pages by selector, attribute, and optional conversion

Fields:

Name Type Description
Name string Name of the output field populated from this extraction rule. Names must be unique within one scrape request or Traversal level
Selector string Selector used to read matching elements. Use ‘CSS: ' or 'XPATH: '. Use '*' to target the whole page; when '*' is used, AttributeName is ignored
AttributeName string or null Optional. Attribute read from each matched element. Use ‘val’, or omit/pass null or empty, to return inner text
Convert string or null Optional. Conversion applied to scraped values. Use ‘md()’ for Markdown or ‘sr()’ for main readable content using Mozilla Readability. Omitted or null applies no conversion

TraversalBranch

Describes a child Traversal branch and the link extraction rule used to reach it from the current level

TraversalBranch inherits the fields of TraversalConfig and adds this field:

Name Type Description
EntryRule CrawlParams Link extraction rule that discovers pages belonging to this child branch

Behavior

Read-only: no; idempotent: yes; destructive: yes; open-world: no.

Please rotate your device to landscape mode

This documentation is specifically designed with a wider layout to provide a better reading experience for code examples, tables, and diagrams.
Rotating your device horizontally ensures you can see everything clearly without excessive scrolling or resizing.

Return to Web Data Source Home