reindex

Describes to an AI agent how to reenroll a web resource into an index for further retrieval

How to call

In supported clients the command syntax can vary. If the WDS MCP server is registered as wds, use:

Arguments

Returned Prompt Text

The block below contains the complete prompt returned to the MCP client. Runtime placeholders map to arguments as follows:

• <url> — url

ROLE:
Act as a web developer with experience in designing multi-page web resources, including paged data, templating, and expertise in pure CSS as well as in modern CSS frameworks: Bootstrap, Tailwind CSS, Foundation, Bulma, UIkit, Tachyons, Materialize, Metro, Spectre.css, Milligram, Pure.css, Skeleton, NES.css, W3.CSS, Picnic CSS, Basscss, Shoelace, etc.
Your experience includes working with web resources that have complex structures, such as those with paged data, hierarchical menus, and leaf pages with data accessible from various navigation elements. You are skilled in identifying and extracting data fields from web pages using CSS selectors and XPath expressions.

OBJECTIVE:
Create or reuse a crawl job for TARGET_URL, configure its content scope for retrieval indexing, start it once, and verify successful enrollment.

RESTRICTIONS:
    -   The agent must use the exact values returned by MCP tools as inputs for later steps. Do not rewrite or guess job names, URLs, run numbers, cursors, or selectors.
    -   MCP structured tool results are serialized as lower-camel JSON fields. Use returned field names such as runNum, completeDateUtc, failedDownloadTasks, data, and dataCursor when reading tool results.
    -   The agent must perform all steps of the PROCEDURE in order, without skipping or parallel execution.
    -   Call MCP tools with named arguments matching the tool parameter names.
    -   WdsFetchChunked returns the download task ID as a string. It does not return raw page text.
    -   Use WdsGetChunk(taskId=taskId, startIndex=startIndex, length=length) to read task content in chunks. Do not use shell/file tools to recover hidden or truncated content from MCP output.
    -   Resolve relative links against the current page URL before passing them to WdsFetchChunked.
    -   To change a JobConfig, read the current object with WdsGetJobConfig when it exists, create or modify the complete object in memory, then persist it with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg). Preserve all unrelated existing properties.
    -   When setting nested JobConfig properties, create missing nested objects or arrays before setting their fields.
    -   To change a TraversalConfig, create or modify the complete object in memory, then persist it with WdsUpsertTraversalConfig(jobName=jobName, traversalConfig=traversalConfig).
    -   WdsRunTraversal and WdsTraversalAll return TraversalRunResult, not crawl data. Use runNum with WdsGetTraversalRunInfo, WdsGetTraversalRunErrors, and WdsGetTraversalRunData.
    -   On any MCP tool error, the agent must STOP WITH REASON and surface the error response.

GLOBAL CONSTANTS:
    TARGET_URL = '<url>'
    RESTART_MODE = 'FromScratch'

DEFINITIONS:
    GetDataRootSelectors(jobName, url)
        Objective:
        Identify the core content (payload) areas of a web page and the navigation sections that lead to similar pages, then derive stable CSS selectors that represent those payload regions.
        Procedure:
	            1.  Create a RESULT string 'CSS: '.
		        2.	Fetch the HTML:

            -   taskId = WdsFetchChunked(jobName=jobName, url=url)
            -   REPEAT:
                -   contentResult = WdsGetChunk(taskId=taskId, startIndex=0, length=1024)
                -   If contentResult is not null: BREAK
                -   Continue polling WdsGetChunk with the same arguments until contentResult is not null.
            -   pageHtml = contentResult.contentSpan if it is not null; otherwise ''
            -   contentLength = contentResult.contentLength if it is not null; otherwise length(pageHtml)
            -   If more unseen content is required for the current analysis:
                -   REPEAT:
                    -   startIndex = length(pageHtml)
                    -   If startIndex >= contentLength: BREAK
                    -   contentResult = WdsGetChunk(taskId=taskId, startIndex=startIndex, length=1024)
                    -   If contentResult is null: continue polling WdsGetChunk with the same arguments until contentResult is not null
                    -   Append contentResult.contentSpan to pageHtml
                -   UNTIL enough relevant content has been inspected or startIndex >= contentLength
            -   Analyze only the content present in pageHtml. If required evidence is in an unread span, read the next span before making conclusions.
		        3.	Analyze the structure:
	                -   Parse pageHtml into its top-level sections: e.g., <header>, <nav>, <main>, <article>, <aside>, <footer>, <div> with a class that most likely means it is a payload section.
                -   Classify each section as either:
	                -   Navigation section — the most top level tags that contain menus, lists, or link groups.
	                -   Payload section — the most top level tags that contain the main informational content of the page.
	        4.	Extract payload selectors:
                -   For every payload section, compute the minimal stable CSS selector that uniquely identifies that element and add it to the RESULT string as a comma-separated selector list with no trailing comma (e.g., CSS: dataSelector1, dataSelector2).
	        5.	Expand via navigation links:
	                -   For navigation sections that likely point to other content pages (exclude ads, external, or third-party links):
		                -   Pick one representative link (<a>) from pageHtml, resolve relative URLs against url, then fetch its HTML:

            -   taskId = WdsFetchChunked(jobName=jobName, url=linkedUrl)
            -   REPEAT:
                -   contentResult = WdsGetChunk(taskId=taskId, startIndex=0, length=1024)
                -   If contentResult is not null: BREAK
                -   Continue polling WdsGetChunk with the same arguments until contentResult is not null.
            -   linkedPageHtml = contentResult.contentSpan if it is not null; otherwise ''
            -   contentLength = contentResult.contentLength if it is not null; otherwise length(linkedPageHtml)
            -   If more unseen content is required for the current analysis:
                -   REPEAT:
                    -   startIndex = length(linkedPageHtml)
                    -   If startIndex >= contentLength: BREAK
                    -   contentResult = WdsGetChunk(taskId=taskId, startIndex=startIndex, length=1024)
                    -   If contentResult is null: continue polling WdsGetChunk with the same arguments until contentResult is not null
                    -   Append contentResult.contentSpan to linkedPageHtml
                -   UNTIL enough relevant content has been inspected or startIndex >= contentLength
            -   Analyze only the content present in linkedPageHtml. If required evidence is in an unread span, read the next span before making conclusions.
		                -   Repeat the structure analysis on linkedPageHtml and collect additional payload selectors found on that page until all new CSS selectors from a page are present in the RESULT string. This indicates that the layouts of all pages have been analyzed, and the Procedure is complete.
	        6.	Return the RESULT string.
        Restrictions:
        -   Fetch and analyse up to five representative internal pages in addition to the provided main page URL. If fewer internal links are available, document how many were analysed and continue.
        -   The RESULT string must contain at least one reliable selector gathered from fetched pages. If only one selector is valid, use it and document why broader selectors were rejected.

PROCEDURE (EXECUTE IN EXACT ORDER)
    1. Identify host
    -	Extract authority/host from TARGET_URL -> lowercase, normalized -> tgtHost
    2. Get or create job
    -   jobs = WdsGetJobs()
    -   Find job where job.jobName equals tgtHost (case-insensitive)
    -   If not found:
        -   jobName = tgtHost
        -   Do not call WdsGetJobConfig before creating the job; a missing job is expected on this path.
        -   Determine jobType:
            -   'Intranet' if host is single-label or ends with .local, .internal, .svc.cluster.local, or lacks a TLD
            -   Otherwise 'Internet'
        -   Create jobCfg as a complete JobConfig:
            -   startUrls = [TARGET_URL]
            -   type = jobType
            -   restart.jobRestartMode = 'Continue'
        -   Persist jobCfg with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg)
        -   jobCfg = WdsGetJobConfig(jobName=jobName)
    -   If found:
        -   jobName = job.jobName
        -   jobCfg = WdsGetJobConfig(jobName=jobName)
    3. Configure job for selector discovery
       -    Update jobCfg in memory, preserving all unrelated properties:
            -   jobCfg.restart.jobRestartMode = RESTART_MODE
            -   jobCfg.retrieval.enrollInIndex = false
       -    Persist jobCfg with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg)
       -    jobCfg = WdsGetJobConfig(jobName=jobName)
    4. Define content scope and configure enrollment
       -    dataSelectors = GetDataRootSelectors(jobName, TARGET_URL)
       -    pathPattern = '/*'
       -    Update jobCfg in memory, preserving all unrelated properties:
            -   jobCfg.retrieval.enrollInIndex = true
            -   jobCfg.retrieval.force = true
            -   jobCfg.retrieval.contentScopes = [{ pathPattern: pathPattern, selector: dataSelectors }]
       -    Persist jobCfg with WdsUpsertJobConfig(jobName=jobName, jobConfig=jobCfg)
    5. Configure crawling
       -    Create crawlCfg as a complete TraversalConfig:
            -   name = '/'
            -   crawlParams = [{ selector: '*', attributeName: null }]
            -   scrapeParams = [{ name: 'Content', selector: '*', attributeName: null }]
       -    Persist crawlCfg with WdsUpsertTraversalConfig(jobName=jobName, traversalConfig=crawlCfg)
    6. Start crawling
    -   traversalRunResult = WdsRunTraversal(jobName=jobName)
    -   runNum = traversalRunResult.runNum
    7. Validate crawl completion
    -   Wait for the traversal run to complete:
        -   maxPolls = 60
        -   pollIntervalSec = 1
        -   pollCount = 0
        -   lastTraversalRunInfo = null
        -   REPEAT while pollCount < maxPolls:
            -   pollCount = pollCount + 1
            -   traversalRunInfo = WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum)
            -   lastTraversalRunInfo = traversalRunInfo
            -   If traversalRunInfo.completeDateUtc is not null:
                -   BREAK
            -   Wait pollIntervalSec seconds before the next poll.
            -   Continue polling with WdsGetTraversalRunInfo(jobName=jobName, runNum=runNum) until traversalRunInfo.completeDateUtc is not null or pollCount reaches maxPolls.
        -   If lastTraversalRunInfo.completeDateUtc is null:
            -   STOP WITH REASON because traversal run did not report completion after maxPolls polls; include runNum and lastTraversalRunInfo in the reason.
    -   traversalRunErrors = WdsGetTraversalRunErrors(jobName=jobName, runNum=runNum)
    -   If traversalRunErrors.failedDownloadTasks is NOT empty:
        -   For each downloadTask in traversalRunErrors.failedDownloadTasks:
            -   For each failedDownloadTask in downloadTask.failedDownloadTasks:
                -   status = WdsGetDownloadTaskStatus(taskId=failedDownloadTask.id)
                -   If status.result is null:
                    -   Collect status.url, null, null
                -   If status.result is not null:
                    -   Collect status.url, status.result.httpStatusCode, status.result.reasonPhrase
        -   Output a table: | URL | HttpStatusCode | ReasonPhrase |
    -   Proceed with successfully gathered data
    8. Complete
       -    Inform the user that enrollment completed.