Overview

The WDS server is built from eight service projects. It can run as one in-process service or as a distributed service set.

Service inventory

Docs and Playground are auxiliary deployment containers, not WDS server services. Compose and Helm deployments may include them for offline documentation and repeatable examples; see Auxiliary containers.

Single-service mode

Solidstack runs the Dapi, Crawler, Datakeeper, Idealer, Jober, Scraper, and Retriever implementations in one process. It is suited to evaluation and small workloads, but its components cannot be scaled or restarted independently. The current Solidstack feature provider does not enable Scheduling. See Plans for feature availability.

Multi-service mode

Multi-service deployments run Dapi, Crawler, Datakeeper, Scraper, Idealer, Retriever, and Jober as separate processes. This topology supports independent scaling, health monitoring, resource allocation, and failure isolation. It is available with the Business plan.

Dapi is the public entry point. It stores job configurations in MongoDB and calls the other services over gRPC-Web. Crawler registers with Datakeeper Resource Manager; Datakeeper, Scraper, and Retriever use Idealer; and Jober coordinates Dapi, Datakeeper, and Scraper for Traversal runs.

FIPS images

FIPS stands for Federal Information Processing Standards, a set of U.S. government standards published by NIST. WDS FIPS images support deployments subject to the FIPS 140-3 standard, which defines security requirements for cryptographic modules. Compliance requires a host running in FIPS mode and validated cryptographic modules used in accordance with their applicable security policies.

Use FIPS images when your deployment must use FIPS-validated cryptography to meet a regulatory, contractual, or organizational security requirement—for example, in U.S. federal systems or contractor environments where that requirement applies. Regular images are suitable when no such requirement applies.

FIPS image variants are available for all services. Each service page above lists its regular and FIPS images. To select a FIPS image, append -fips to the image repository name and keep the same version tag.

Note: Regular service images are multi-platform. The -fips service images are built only for linux/amd64.

Note: For WDS FIPS images the host kernel must also have FIPS mode enabled.

Runtime dependencies

WDS lets you choose between a ready-to-run container and a deployment backed by infrastructure you manage. The supporting services depend on the features you use:

• Start with MongoDB. WDS uses MongoDB to store its data and service state. In multi-service deployments, every service except Crawler connects to MongoDB; Solidstack shares one MongoDB configuration across its built-in components.
• Choose your cache storage at deployment time. Datakeeper can cache downloaded pages in its primary MongoDB database (the default), a separate MongoDB database, or S3-compatible storage. Choose the backend before you start; S3-compatible storage is recommended for production.
• Add search and AI retrieval. Retrieval requires MongoDB Atlas Search support, either in the primary database or, for Retriever, a separate search database. Vector search also needs an HTTP embedding service to turn content and queries into vectors. The same search and embedding requirements apply when Retrieval runs inside Solidstack.
• Simplify setup with an all-in-one image. Solidstack AIO images bundle MongoDB. The FTS and FTVS variants include MongoDB Atlas Local for search, and the FTVS variant also includes Ollama with a preloaded embedding model.