Better RAG Search Starts with Better Chunks

Better RAG search starts with better chunks.

How web content is prepared shapes what your retrieval system can find—and the context it can return.

Web Data Source (WDS) prepares that content in two stages:

1. HTML → Markdown
   WDS converts page content into Markdown, removing presentation markup and reducing layout noise before indexing.

2. Markdown → Semantic chunks
   WDS groups related content into meaningful chunks while respecting the configured maximum tokens per indexed chunk.

The goal is to preserve useful context within a controlled size, helping retrieval return passages that make sense when read on their own.

With MaxTokensPerChunk, you control that size limit. The default is 512 tokens per indexed chunk.

For documentation, policies, and product information, this approach helps prepare cleaner, more coherent context for search and downstream AI answers.

Clean the content. Preserve the meaning. Keep the size under control.

Explore WDS Retrieval configuration