Deploying WDS API Server in Docker Compose

Recommended for evaluation and development: run the API Server plus dependencies locally via Compose. The API binds to port 2807 at http://localhost:2807. See Plans for feature availability.

To make use of this deployment type, docker-compose needs to be installed locally. Use the following command to check if it’s already installed:

docker compose version

the result should be like this:

Docker Compose version <version>

If the result shows that docker-compose is not installed, please review the installation documentation on its official website

Auxiliary containers

The Compose topologies include two optional companion containers that are not WDS server services:

Deployment Options

Taking into account that the API Server requires third-party components, there are several deployment options:

FIPS: For the OEM and BOX options, replace each webdatasource/<service>:<tag> core service image with its corresponding webdatasource/<service>-fips:<tag> image to run the core services in FIPS-compliant mode. FIPS images are available only for linux/amd64; Playground and Docs are unaffected.

This deployment option uses Solidstack AIO FTVS to bundle MongoDB Atlas Local, Ollama, and the embeddinggemma model in the Solidstack container. Retrieval is enabled in FullTextAndVector mode without external database or embedding services. Its bundled credentials are for evaluation only.


services:
  wds.solidstack:
    image: docker.io/webdatasource/solidstack-aio-ftvs:v3.0.0
    restart: unless-stopped
    hostname: solidstack
    environment:
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - JOB_TYPES=intranet
    ports:
      - 0.0.0.0:2807:8080

  wds.playground:
    image: docker.io/webdatasource/playground:v3.0.0
    restart: unless-stopped
    hostname: playground
    ports:
      - 0.0.0.0:2808:80

  wds.docs:
    image: docker.io/webdatasource/docs:3.0.3
    restart: unless-stopped
    ports:
      - 0.0.0.0:2809:80
This deployment option runs the distributed WDS services and auxiliary containers without third-party infrastructure. Replace each database, search, cache, and embedding placeholder with an external service that is reachable from the Compose network. It includes Dapi, Crawler, Datakeeper, Idealer, Scraper, Retriever, and Jober, plus Playground and Docs.

Replace the value of the EMBEDDING_SERVICE_URL with a URL to an embedding service. For evaluation purposes, this service can be hosted locally:

If it’s a SaaS, the API Key can be set in the EMBEDDING_SERVICE_API_KEY environment variable

By default, system DB (MongoDB) is used to cache web pages. This behavior can be changed by providing the wds.datakeeper service with a CACHE_CONNECTION_STRING.

IMPORTANT! MongoDB connection strings must contain a database name. The names might be the same as values in the example connection strings or others. IMPORTANT! This feature is available starting from the Business Plan


services:
  wds.crawler:
    image: docker.io/webdatasource/crawler:v3.0.0
    restart: unless-stopped
    hostname: crawler
    environment:
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SERVICE_HOST=crawler
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - LICENSE_KEY=[LICENSE]

  wds.dapi:
    image: docker.io/webdatasource/dapi:v3.0.0
    restart: unless-stopped
    hostname: dapi
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Dapi?appName=<cluster>&readPreference=secondary
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - IDEALER_ORIGIN=http://idealer
      - JOBER_ORIGIN=http://jober
      - RETRIEVER_ORIGIN=http://retriever
      - JOB_TYPES=intranet
      - LICENSE_KEY=[LICENSE]
    ports:
      - 0.0.0.0:2807:8080

  wds.jober:
    image: docker.io/webdatasource/jober:v3.0.0
    restart: unless-stopped
    hostname: jober
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Jober?appName=<cluster>&readPreference=secondary
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - DAPI_ORIGIN=http://dapi
      - LICENSE_KEY=[LICENSE]

  wds.datakeeper:
    image: docker.io/webdatasource/datakeeper:v3.0.0
    restart: unless-stopped
    hostname: datakeeper
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Datakeeper?appName=<cluster>&readPreference=secondary
      - CACHE_CONNECTION_STRING=s3://<access-key>:<secret-key>@<host>:<port>/<bucket>?ssl=true
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]

  wds.idealer:
    image: docker.io/webdatasource/idealer:v3.0.0
    restart: unless-stopped
    hostname: idealer
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Idealer?appName=<cluster>&readPreference=secondary
      - LICENSE_KEY=[LICENSE]

  wds.scraper:
    image: docker.io/webdatasource/scraper:v3.0.0
    restart: unless-stopped
    hostname: scraper
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Scraper?appName=<cluster>&readPreference=secondary
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]

  wds.retriever:
    image: docker.io/webdatasource/retriever:v3.0.0
    restart: unless-stopped
    hostname: retriever
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Retriever?appName=<cluster>&readPreference=secondary
      - SEARCH_DB_CONNECTION_STRING=mongodb://<user>:<password>@<host>:<port>/Retriever?authSource=admin&directConnection=true
      - EMBEDDING_SERVICE_URL=http://<host>:<port>/api/embed
      - IDEALER_ORIGIN=http://idealer
      - SEARCH_MODE=FullTextAndVector
      - LICENSE_KEY=[LICENSE]

  wds.playground:
    image: docker.io/webdatasource/playground:v3.0.0
    restart: unless-stopped
    hostname: playground
    ports:
      - 0.0.0.0:2808:80

  wds.docs:
    image: docker.io/webdatasource/docs:3.0.3
    restart: unless-stopped
    ports:
      - 0.0.0.0:2809:80
This deployment option is intended to contain the distributed WDS services, bundled MongoDB/search/cache infrastructure, and auxiliary containers. It runs one instance of each service and uses evaluation-only credentials.

MongoDB Atlas Local supplies the system and Atlas Search databases. A separate Ollama container supplies the embedding service. A one-shot initializer downloads embeddinggemma:latest into the persistent embedding-models volume before Retriever starts, and Retriever calls the internal http://embedding-service:11434/api/embed endpoint.

IMPORTANT! This feature is available starting from the Business Plan


services:
  mongodb:
    image: docker.io/mongodb/mongodb-atlas-local:8.3.8
    restart: always
    hostname: mongodb
    environment:
      - MONGODB_INITDB_ROOT_USERNAME=root
      - MONGODB_INITDB_ROOT_PASSWORD=TestPassword
    volumes:
      - mongodb-data:/data/db
      - mongodb-config:/data/configdb
    healthcheck:
      test: ["CMD-SHELL", "mongosh --host 127.0.0.1 --username root --password TestPassword --authenticationDatabase admin --quiet --eval 'quit(db.adminCommand({ ping: 1 }).ok ? 0 : 2)'"]
      interval: 5s
      timeout: 3s
      retries: 40

  minio:
    image: docker.io/bitnami/minio:2024
    restart: always
    hostname: minio
    environment:
      - MINIO_DEFAULT_BUCKETS=pages-cache
      - MINIO_ROOT_USER=TestAccessKey
      - MINIO_ROOT_PASSWORD=TestSecretKey
      - MINIO_FORCE_NEW_KEYS=no

  embedding-service:
    image: docker.io/ollama/ollama:latest
    restart: always
    hostname: embedding-service
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_MODELS=/root/.ollama/models
    volumes:
      - embedding-models:/root/.ollama
    healthcheck:
      test: ["CMD-SHELL", "OLLAMA_HOST=http://127.0.0.1:11434 ollama show embeddinggemma:latest >/dev/null 2>&1"]
      interval: 5s
      timeout: 3s
      retries: 40

  embedding-model-init:
    image: docker.io/ollama/ollama:latest
    restart: "no"
    depends_on:
      embedding-service:
        condition: service_started
    environment:
      - OLLAMA_HOST=http://embedding-service:11434
    entrypoint:
      - /bin/sh
      - -ec
      - until ollama list >/dev/null 2>&1; do sleep 1; done; ollama show embeddinggemma:latest >/dev/null 2>&1 || ollama pull embeddinggemma:latest
 
  wds.crawler:
    image: docker.io/webdatasource/crawler:v3.0.0
    restart: always
    hostname: crawler
    environment:
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SERVICE_HOST=crawler
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - LICENSE_KEY=[LICENSE]
  
  wds.datakeeper:
    image: docker.io/webdatasource/datakeeper:v3.0.0
    restart: always
    hostname: datakeeper
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Datakeeper?authSource=admin&directConnection=true
      - CACHE_CONNECTION_STRING=s3://TestAccessKey:TestSecretKey@minio:9000/pages-cache?ssl=false
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]

  wds.idealer:
    image: docker.io/webdatasource/idealer:v3.0.0
    restart: always
    hostname: idealer
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Idealer?authSource=admin&directConnection=true
      - LICENSE_KEY=[LICENSE]
  
  wds.scraper:
    image: docker.io/webdatasource/scraper:v3.0.0
    restart: always
    hostname: scraper
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Scraper?authSource=admin&directConnection=true
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]
  
  wds.dapi:
    image: docker.io/webdatasource/dapi:v3.0.0
    restart: always
    hostname: dapi
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Dapi?authSource=admin&directConnection=true
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - IDEALER_ORIGIN=http://idealer
      - JOBER_ORIGIN=http://jober
      - RETRIEVER_ORIGIN=http://retriever
      - JOB_TYPES=intranet
      - LICENSE_KEY=[LICENSE]
    ports:
      - 2807:8080

  wds.jober:
    image: docker.io/webdatasource/jober:v3.0.0
    restart: always
    hostname: jober
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Jober?authSource=admin&directConnection=true
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - DAPI_ORIGIN=http://dapi
      - LICENSE_KEY=[LICENSE]

  wds.retriever:
    image: docker.io/webdatasource/retriever:v3.0.0
    restart: always
    hostname: retriever
    depends_on:
      mongodb:
        condition: service_healthy
      embedding-model-init:
        condition: service_completed_successfully
      embedding-service:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Retriever?authSource=admin&directConnection=true
      - SEARCH_DB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Retriever?authSource=admin&directConnection=true
      - EMBEDDING_SERVICE_URL=http://embedding-service:11434/api/embed
      - IDEALER_ORIGIN=http://idealer
      - SEARCH_MODE=FullTextAndVector
      - LICENSE_KEY=[LICENSE]
  
  wds.playground:
    image: docker.io/webdatasource/playground:v3.0.0
    restart: always
    hostname: playground
    ports:
      - 2808:80
  
  wds.docs:
    image: docker.io/webdatasource/docs:3.0.3
    restart: always
    ports:
      - 2809:80

volumes:
  mongodb-data:
  mongodb-config:
  embedding-models:

Running Docker Compose

After an appropriate deployment option is selected and a docker-compose configuration has been copied to a file (e.g., wds-docker-compose.yml), there are two types of scripts to run docker-compose:

NOTE_ If another version is currently running, just replace the old docker-compose configuration with the new one (of the same option) and execute the selected script. The only services with changed versions will be recreated. If the new version is of a different option, execute the docker compose down command beforehand.

set COMPOSE_FILE=wds-docker-compose.yml && docker compose up -d

export COMPOSE_FILE=wds-docker-compose.yml && docker compose up -d

Please rotate your device to landscape mode

This documentation is specifically designed with a wider layout to provide a better reading experience for code examples, tables, and diagrams.
Rotating your device horizontally ensures you can see everything clearly without excessive scrolling or resizing.

Return to Web Data Source Home