Retriever Service
Retriever provides indexing and search for job-scoped retrieval and cross-job retrieval. It enrolls crawl results, runs full-text and vector searches, obtains embeddings from a configurable HTTP service, and tracks each job’s enrollment state.
Release image: webdatasource/retriever:v3.0.0
Configuration
| Name | Required | Default | Meaning |
|---|---|---|---|
MONGODB_CONNECTION_STRING |
Yes | None | MongoDB connection string for Retriever state. Include a database name unless MONGODB_DATABASE_NAME is set. |
MONGODB_DATABASE_NAME |
No | Database from the connection string | Overrides the primary MongoDB database name. |
IDEALER_ORIGIN |
Yes | None | Origin of Idealer; Retriever appends port 8082 for gRPC-Web. |
SEARCH_MODE |
No | FullText |
Search behavior: FullText, Vector, or FullTextAndVector. |
SEARCH_DB_CONNECTION_STRING |
No | Primary MongoDB connection | Selects a dedicated MongoDB Atlas Search database. |
SEARCH_DB_DATABASE_NAME |
No | Database from the search connection string | Overrides the dedicated search database name. |
EMBEDDING_SERVICE_URL |
For vector modes | None | HTTP endpoint used to create embeddings. |
EMBEDDING_SERVICE_API_KEY |
No | None | Bearer token sent to the embedding service. |
EMBEDDING_SERVICE_REQUEST_TEMPLATE |
No | {'model':'embeddinggemma','input':null} |
JSON request template sent to the embedding service. |
EMBEDDING_SERVICE_CONTENT_JSON_PATH |
No | $.input |
JSONPath whose value is replaced with the input array. |
EMBEDDING_SERVICE_RESULT_JSON_PATH |
No | $.embeddings |
JSONPath used to extract the returned embeddings array. |
EMBEDDING_VECTORS_LENGTH |
No | 768 |
Number of dimensions stored in the MongoDB Atlas vector index. |
LICENSE_KEY |
Yes | None | WDS license key with the Retrieval feature. See Plans. |
HEALTH_PROBE_LOG_LEVEL |
No | Debug |
Log level used for /health and /ready request messages. |
MIN_LOG_LEVEL |
No | Info |
Minimum application log level. |
Endpoints and dependencies
Port 8080 serves /health and /ready. Port 8082 serves Retriever gRPC-Web over HTTP/1.1. Retriever uses its primary MongoDB database, connects to Idealer, and optionally connects to a separate MongoDB Atlas Search database and an HTTP embedding service.
Search database
Retriever currently uses MongoDB Atlas Search for full-text and vector indexes. If SEARCH_DB_CONNECTION_STRING is omitted, the primary MongoDB deployment must support the selected Atlas Search mode. Vector modes also require a reachable embedding service and matching EMBEDDING_VECTORS_LENGTH.