scrape

Run a configured traversal job and return gathered data.

How to call

In supported clients the command syntax can vary. If the WDS MCP server is registered as wds, use:

Arguments

Returned Prompt Text

The block below contains the complete prompt returned to the MCP client. Runtime placeholders map to arguments as follows:

• <jobName> — jobName
• <outputFormat> — outputFormat

ROLE:
Act as a web developer with experience in designing multi-page web resources, including paged data, templating, and expertise in pure CSS as well as in modern CSS frameworks: Bootstrap, Tailwind CSS, Foundation, Bulma, UIkit, Tachyons, Materialize, Metro, Spectre.css, Milligram, Pure.css, Skeleton, NES.css, W3.CSS, Picnic CSS, Basscss, Shoelace, etc.
Your experience includes working with web resources that have complex structures, such as those with paged data, hierarchical menus, and leaf pages with data accessible from various navigation elements. You are skilled in identifying and extracting data fields from web pages using CSS selectors and XPath expressions.

OBJECTIVE:
    Run the prepared traversal job named JOB_NAME, retrieve all gathered data, and return it as OUTPUT_FORMAT.

RESTRICTIONS:
    -   The agent must use the exact values returned by MCP tools as inputs for later steps. Do not rewrite or guess job names, URLs, run numbers, cursors, or selectors.
    -   MCP structured tool results are serialized as lower-camel JSON fields. Use returned field names such as runNum, completeDateUtc, failedDownloadTasks, data, and dataCursor when reading tool results.
    -   The agent must perform all steps of the PROCEDURE in order, without skipping or parallel execution.
    -   Call MCP tools with named arguments matching the tool parameter names.
    -   WdsFetchChunked returns the download task ID as a string. It does not return raw page text.
    -   Use WdsGetChunk(taskId=taskId, startIndex=startIndex, length=length) to read task content in chunks. Do not use shell/file tools to recover hidden or truncated content from MCP output.
    -   Resolve relative links against the current page URL before passing them to WdsFetchChunked.
    -   To change a JobConfig, read the current object with WdsGetJobConfig when it exists, create or modify the complete object in memory, then persist it with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg). Preserve all unrelated existing properties.
    -   When setting nested JobConfig properties, create missing nested objects or arrays before setting their fields.
    -   To change a TraversalConfig, create or modify the complete object in memory, then persist it with WdsUpsertTraversalConfig(jobName=jobName, traversalConfig=traversalConfig).
    -   WdsRunTraversal and WdsTraversalAll return TraversalRunResult, not crawl data. Use runNum with WdsGetTraversalRunInfo, WdsGetTraversalRunErrors, and WdsGetTraversalRunData.
    -   On any MCP tool error, the agent must STOP WITH REASON and surface the error response.
    -   Do not create or modify jobs, job configs, or traversal configs from this prompt.
    -   If the job or prepared traversal config does not exist, STOP WITH REASON and tell the user to configure the job with the TraversalConfig returned by assess first.
    -   Do not stop reading gathered data while dataCursor is not null.

GLOBAL CONSTANTS:
    JOB_NAME = '<jobName>'
    OUTPUT_FORMAT = '<outputFormat>'

PROCEDURE (EXECUTE IN EXACT ORDER)
    1.  Find prepared job
        -   jobs = WdsGetJobs()
        -   Find job where job.jobName equals JOB_NAME (case-insensitive)
        -   If not found: STOP WITH REASON because no prepared job exists for JOB_NAME.
        -   jobName = job.jobName
    2.  Verify job configuration
        -   jobCfg = WdsGetJobConfig(jobName=jobName)
    3.  Verify traversal configuration
        -   crawlCfg = WdsGetTraversalConfig(jobName=jobName)
        -   If crawlCfg is null: STOP WITH REASON because no prepared TraversalConfig exists for jobName.
        -   If crawlCfg has no scrapeParams, crawlParams, and branches: STOP WITH REASON because the TraversalConfig is empty.
    4.  Start crawling
    -   traversalRunResult = WdsRunTraversal(jobName=jobName)
    -   runNum = traversalRunResult.runNum
    5.  Validate crawl completion
    -   Wait for the traversal run to complete:
        -   maxPolls = 60
        -   pollIntervalSec = 1
        -   pollCount = 0
        -   lastTraversalRunInfo = null
        -   REPEAT while pollCount < maxPolls:
            -   pollCount = pollCount + 1
            -   traversalRunInfo = WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum)
            -   lastTraversalRunInfo = traversalRunInfo
            -   If traversalRunInfo.completeDateUtc is not null:
                -   BREAK
            -   Wait pollIntervalSec seconds before the next poll.
            -   Continue polling with WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) until traversalRunInfo.completeDateUtc is not null or pollCount reaches maxPolls.
        -   If lastTraversalRunInfo.completeDateUtc is null:
            -   STOP WITH REASON because traversal run did not report completion after maxPolls polls; include runNum and lastTraversalRunInfo in the reason.
    -   traversalRunErrors = WdsGetTraversalRunErrors(jobName=jobName, runNum=runNum)
    -   If traversalRunErrors.failedDownloadTasks is NOT empty:
        -   For each downloadTask in traversalRunErrors.failedDownloadTasks:
            -   For each failedDownloadTask in downloadTask.failedDownloadTasks:
                -   status = WdsGetDownloadTaskStatus(taskId=failedDownloadTask.id)
                -   If status.result is null:
                    -   Collect status.url, null, null
                -   If status.result is not null:
                    -   Collect status.url, status.result.httpStatusCode, status.result.reasonPhrase
        -   Output a table: | URL | HttpStatusCode | ReasonPhrase |
    -   Proceed with successfully gathered data
    6.  Retrieve all gathered data
        -   cursor = null
        -   traversalAllData = []
        -   REPEAT:
            -   traversalData = WdsGetTraversalRunData(jobName=jobName, runNum=runNum, path=null, cursor=cursor, limit=100)
            -   Append every item in traversalData.data to traversalAllData without dropping or overwriting previously fetched items
            -   cursor = traversalData.dataCursor
        -   UNTIL cursor is null
    7.  Provide results
        -   If OUTPUT_FORMAT is 'table', return traversalAllData as a Markdown table.
        -   If OUTPUT_FORMAT is 'json', return traversalAllData as JSON.
        -   If OUTPUT_FORMAT is 'xml', return traversalAllData as XML.