Deploying WDS API Server in Air-Gapped Environments

Run WDS in air‑gapped networks by mirroring Docker images to a private registry and pointing deployments to it. See Plans for feature availability.

It’s required to copy necessary docker images in a private docker registry available from an air-gapped environment to deploy the system there. The image set differs by deployment option.
Additionally to the system services the third-party components should be in the private docker registry or provided as a service.

Use the following form to specify a particular private docker registry address. This value will be applied to all script examples and docker-compose configurations so they can be copied and used right from this page.

Script Examples

The following examples show how to copy the images currently defined by the release globals to a private docker registry.
There are two scripts:

archive-wds-images - downloads all required docker images to archive them into a single wds-docker-images.tar
push-wds-images - pushes all (or only those required for a specific deployment option) WDS images to a private docker registry

The scripts include every release WDS image, including all Solidstack AIO variants, MongoDB Atlas Local 8.3.8, Ollama, and Jober. The AIO images are mirrored for Docker and Docker Compose deployments; Helm uses the standard Solidstack image. The scripts are complete for MINI and for the Business options when their documented external dependencies are supplied. BOX Business includes Atlas Search and an Ollama embedding service. Because the embeddinggemma model is stored in a volume rather than in the Ollama image, prepare and transfer the embedding-models volume before disconnecting the target environment; the first-start initializer otherwise needs internet access to download the model.

archive-wds-images.bat


@echo off
setlocal enabledelayedexpansion

if "%~1"=="" (
  echo Error: No output tar archive path provided.
  echo Usage: %~0 ^<path-to-wds-images.tar^>
  exit /b 1
)

set ARCHIVE_PATH=%~1

REM List of images you want to bundle into the tar
set IMAGES=^
mongodb/mongodb-atlas-local:8.3.8 ^
bitnami/minio:2024 ^
ollama/ollama:latest ^
webdatasource/solidstack:v3.0.0 ^
webdatasource/solidstack-fips:v3.0.0 ^
webdatasource/solidstack-aio:v3.0.0 ^
webdatasource/solidstack-aio-fts:v3.0.0 ^
webdatasource/solidstack-aio-ftvs:v3.0.0 ^
webdatasource/crawler:v3.0.0 ^
webdatasource/crawler-fips:v3.0.0 ^
webdatasource/datakeeper:v3.0.0 ^
webdatasource/datakeeper-fips:v3.0.0 ^
webdatasource/idealer:v3.0.0 ^
webdatasource/idealer-fips:v3.0.0 ^
webdatasource/scraper:v3.0.0 ^
webdatasource/scraper-fips:v3.0.0 ^
webdatasource/dapi:v3.0.0 ^
webdatasource/dapi-fips:v3.0.0 ^
webdatasource/jober:v3.0.0 ^
webdatasource/jober-fips:v3.0.0 ^
webdatasource/retriever:v3.0.0 ^
webdatasource/retriever-fips:v3.0.0 ^
webdatasource/playground:v3.0.0 ^
webdatasource/docs:3.0.3

echo Pulling WDS images...
for %%I in (%IMAGES%) do (
  echo Pulling %%I...
  docker pull %%I
  echo.
)

echo Creating a single tar archive with all WDS images at %ARCHIVE_PATH%...
docker save -o %ARCHIVE_PATH% %IMAGES%

echo.
echo Done! Created %ARCHIVE_PATH% containing all WDS images.

endlocal

push-wds-images.bat


@echo off
setlocal enabledelayedexpansion

if "%~1"=="" (
  echo Error: No registry address provided.
  echo Usage: %~0 ^<registry-address^> ^<path-to-wds-docker-images.tar^>
  exit /b 1
)

if "%~2"=="" (
  echo Error: No tar archive path provided.
  echo Usage: %~0 ^<registry-address^> ^<path-to-wds-docker-images.tar^>
  exit /b 1
)

set PRIVATE_REGISTRY=%~1
set ARCHIVE_PATH=%~2
set IMAGES=^
mongodb/mongodb-atlas-local:8.3.8 ^
bitnami/minio:2024 ^
ollama/ollama:latest ^
webdatasource/solidstack:v3.0.0 ^
webdatasource/solidstack-fips:v3.0.0 ^
webdatasource/solidstack-aio:v3.0.0 ^
webdatasource/solidstack-aio-fts:v3.0.0 ^
webdatasource/solidstack-aio-ftvs:v3.0.0 ^
webdatasource/crawler:v3.0.0 ^
webdatasource/crawler-fips:v3.0.0 ^
webdatasource/datakeeper:v3.0.0 ^
webdatasource/datakeeper-fips:v3.0.0 ^
webdatasource/idealer:v3.0.0 ^
webdatasource/idealer-fips:v3.0.0 ^
webdatasource/scraper:v3.0.0 ^
webdatasource/scraper-fips:v3.0.0 ^
webdatasource/dapi:v3.0.0 ^
webdatasource/dapi-fips:v3.0.0 ^
webdatasource/jober:v3.0.0 ^
webdatasource/jober-fips:v3.0.0 ^
webdatasource/retriever:v3.0.0 ^
webdatasource/retriever-fips:v3.0.0 ^
webdatasource/playground:v3.0.0 ^
webdatasource/docs:3.0.3

echo Loading images from tar file: %ARCHIVE_PATH%
docker load -i %ARCHIVE_PATH%

echo.
echo Tagging and pushing wds images to %PRIVATE_REGISTRY%...
for %%I in (%IMAGES%) do (
  echo Tagging %%I as %PRIVATE_REGISTRY%/%%I
  docker tag %%I %PRIVATE_REGISTRY%/%%I

  echo Pushing %PRIVATE_REGISTRY%/%%I
  docker push %PRIVATE_REGISTRY%/%%I
  echo.
)

echo Done! WDS images loaded and pushed to %PRIVATE_REGISTRY%.

endlocal

Download and archive WDS docker images:

IMPORTANT! The current user should have access to the Internet

archive-wds-images.bat wds-docker-images.tar

Load and push WDS docker images to a private registry:

IMPORTANT! The current user should have access and be logged in to the private registry

push-wds-images.bat [registry] wds-docker-images.tar

archive-wds-images.sh


#!/usr/bin/env bash
set -e

if [ -z "$1" ]; then
  echo "Error: No output tar archive path provided."
  echo "Usage: $0 /path/to/wds-docker-images.tar"
  exit 1
fi

ARCHIVE_PATH="$1"

IMAGES=(
    "mongodb/mongodb-atlas-local:8.3.8"
    "bitnami/minio:2024"
    "ollama/ollama:latest"
    "webdatasource/solidstack:v3.0.0"
    "webdatasource/solidstack-fips:v3.0.0"
    "webdatasource/solidstack-aio:v3.0.0"
    "webdatasource/solidstack-aio-fts:v3.0.0"
    "webdatasource/solidstack-aio-ftvs:v3.0.0"
    "webdatasource/crawler:v3.0.0"
    "webdatasource/crawler-fips:v3.0.0"
    "webdatasource/datakeeper:v3.0.0"
    "webdatasource/datakeeper-fips:v3.0.0"
    "webdatasource/idealer:v3.0.0"
    "webdatasource/idealer-fips:v3.0.0"
    "webdatasource/scraper:v3.0.0"
    "webdatasource/scraper-fips:v3.0.0"
    "webdatasource/dapi:v3.0.0"
    "webdatasource/dapi-fips:v3.0.0"
    "webdatasource/jober:v3.0.0"
    "webdatasource/jober-fips:v3.0.0"
    "webdatasource/retriever:v3.0.0"
    "webdatasource/retriever-fips:v3.0.0"
    "webdatasource/playground:v3.0.0"
    "webdatasource/docs:3.0.3"
)

echo "Pulling all images..."
for IMAGE in "${IMAGES[@]}"; do
  echo "Pulling: ${IMAGE}"
  docker pull "${IMAGE}"
  echo
done

echo "Creating a single tar archive with all WDS images at ${ARCHIVE_PATH}..."
docker save -o "${ARCHIVE_PATH}" "${IMAGES[@]}"

echo
echo "Done! Created ${ARCHIVE_PATH} containing all WDS images."

push-wds-images.sh


#!/usr/bin/env bash
set -e

if [ -z "$1" ]; then
  echo "Error: No registry address provided."
  echo "Usage: $0 <registry-address> /path/to/wds-docker-images.tar"
  exit 1
fi

if [ -z "$2" ]; then
  echo "Error: No tar archive provided."
  echo "Usage: $0 <registry-address> /path/to/wds-docker-images.tar"
  exit 1
fi

PRIVATE_REGISTRY="$1"
ARCHIVE_PATH="$2"
IMAGES=(
    "mongodb/mongodb-atlas-local:8.3.8"
    "bitnami/minio:2024"
    "ollama/ollama:latest"
    "webdatasource/solidstack:v3.0.0"
    "webdatasource/solidstack-fips:v3.0.0"
    "webdatasource/solidstack-aio:v3.0.0"
    "webdatasource/solidstack-aio-fts:v3.0.0"
    "webdatasource/solidstack-aio-ftvs:v3.0.0"
    "webdatasource/crawler:v3.0.0"
    "webdatasource/crawler-fips:v3.0.0"
    "webdatasource/datakeeper:v3.0.0"
    "webdatasource/datakeeper-fips:v3.0.0"
    "webdatasource/idealer:v3.0.0"
    "webdatasource/idealer-fips:v3.0.0"
    "webdatasource/scraper:v3.0.0"
    "webdatasource/scraper-fips:v3.0.0"
    "webdatasource/dapi:v3.0.0"
    "webdatasource/dapi-fips:v3.0.0"
    "webdatasource/jober:v3.0.0"
    "webdatasource/jober-fips:v3.0.0"
    "webdatasource/retriever:v3.0.0"
    "webdatasource/retriever-fips:v3.0.0"
    "webdatasource/playground:v3.0.0"
    "webdatasource/docs:3.0.3"
)

echo "Loading images from tar file: ${ARCHIVE_PATH}"
docker load -i "${ARCHIVE_PATH}"

echo
echo "Tagging and pushing WDS images to ${PRIVATE_REGISTRY}..."
for IMAGE in "${IMAGES[@]}"; do
  FINAL_TAG="${PRIVATE_REGISTRY}/${IMAGE}"
  echo "Tagging ${IMAGE} as ${FINAL_TAG}"
  docker tag "${IMAGE}" "${FINAL_TAG}"

  echo "Pushing ${FINAL_TAG}"
  docker push "${FINAL_TAG}"
  echo
done

echo "Done! WDS images loaded and pushed to ${PRIVATE_REGISTRY}."

Download and archive WDS docker images:

IMPORTANT! The current user should have access to the Internet

chmod +x archive-wds-images.sh
./archive-wds-images.sh wds-docker-images.tar

Load and push WDS docker images to a private registry:

IMPORTANT! The current user should have access and be logged in to the private registry

chmod +x push-wds-images.sh
./push-wds-images.sh [registry] wds-docker-images.tar

Deploy to Kubernetes

To deploy a WDS API Server instance to a Kubernetes cluster in an air-gapped environment, the global.registry parameter should be set in the Helm Chart values.
For the other configuration options, see the Helm Chart documentation. Solidstack AIO images are not supported by the Helm chart.

Deploying using Helm CLI

To deploy the WDS API Server using the Helm CLI, run the following commands:

IMPORTANT! Ensure you have access to a Kubernetes cluster.

# Add a repository
helm repo add webdatasource https://webdatasource.github.io/wds.helm

# Update your local repository cache
helm repo update

# Search for the chart to verify it's available
helm search repo webdatasource --version 1.5.0

# Install the chart with standard Solidstack and external MongoDB
helm install wds-server --version 1.5.0 webdatasource/wds-helm-chart \
    --namespace webdatasource --create-namespace \
    --set global.registry="[registry]" \
    --set global.coreServices.databases.mongodb.connectionString="mongodb+srv://<user>:<password>@<host>/WebDataSource?appName=<cluster>&readPreference=secondary"

Deploying using Terraform

IMPORTANT! Ensure you have helm provider configured in your terraform project.

SingleService

Minimal Terraform configuration to deploy WDS API Server:

variable "mongodb_connection_string" {
  description = "MongoDB connection string"
  type        = string
  sensitive   = true
}

variable "registry" {
  description = "Docker registry of WDS API Server service images"
  type        = string
  default     = "[registry]"
}

variable "helm_chart_version" {
  description = "Helm chart version for WDS API Server deployment"
  type        = string
  default     = "1.5.0"
}

variable "create_namespace" {
  description = "Namespace and ingress configuration for WDS API Server deployment. Set `create_namespace` to false to deploy in an existing namespace"
  type        = bool
  default     = true
}

variable "namespace" {
  description = "Namespace for WDS API Server deployment"
  type        = string
  default     = "webdatasource"
}

variable "enable_ingress" {
  description = "Enable or disable ingress for WDS API Server deployment"
  type        = bool
  default     = true
}

# Kubernetes namespace resource
resource "kubernetes_namespace" "webdatasource" {
  count = var.create_namespace ? 1 : 0
  metadata {
    name = var.namespace
  }
}

# Helm release resource
resource "helm_release" "wds-server" {
  name       = "wds-server"
  repository = "https://webdatasource.github.io/wds.helm"
  chart      = "wds-helm-chart"
  version    = var.helm_chart_version

  namespace        = var.namespace
  create_namespace = false

  values = [
    yamlencode({
      global = {
        registry = var.registry
        ingress = {
          enabled = var.enable_ingress
        }
      }
    }),
    sensitive(yamlencode({
      global = {
        coreServices = {
          databases = {
            mongodb = {
              connectionString = var.mongodb_connection_string
            }
          }
        }
      }
    }))
  ]

  depends_on = [
    kubernetes_namespace.webdatasource
  ]
}

MultiService

Terraform configuration to deploy WDS API Server in the MultiService mode with Retrieval:

variable "license_key" {
  description = "WebDataSource license key"
  type        = string
  sensitive   = true
}

variable "embedding_service_url" {
  description = "Embedding service URL"
  type        = string
}

variable "embedding_service_api_key" {
  description = "Embedding service API key"
  type        = string
  sensitive   = true
  default     = ""
}

variable "mongodb_connection_string" {
  description = "MongoDB connection string"
  type        = string
  sensitive   = true
}

variable "registry" {
  description = "Docker registry of WDS API Server service images"
  type        = string
  default     = "[registry]"
}

variable "helm_chart_version" {
  description = "Helm chart version for WDS API Server deployment"
  type        = string
  default     = "1.5.0"
}

variable "create_namespace" {
  description = "Namespace and ingress configuration for WDS API Server deployment. Set `create_namespace` to false to deploy in an existing namespace"
  type        = bool
  default     = true
}

variable "namespace" {
  description = "Namespace for WDS API Server deployment"
  type        = string
  default     = "webdatasource"
}

variable "enable_ingress" {
  description = "Enable or disable ingress for WDS API Server deployment"
  type        = bool
  default     = true
}

# Kubernetes namespace resource
resource "kubernetes_namespace" "webdatasource" {
  count = var.create_namespace ? 1 : 0
  metadata {
    name = var.namespace
  }
}

# Helm release resource
resource "helm_release" "wds-server" {
  name       = "wds-server"
  repository = "https://webdatasource.github.io/wds.helm"
  chart      = "wds-helm-chart"
  version    = var.helm_chart_version

  namespace        = var.namespace
  create_namespace = false

  values = [
    yamlencode({
      global = {
        registry = var.registry
        ingress = {
          enabled = var.enable_ingress
        }
        coreServices = {
          mode = "MultiService"
          search = {
            enabled = true
            mode    = "FullTextAndVector"
            embeddingService = {
              url = var.embedding_service_url
            }
          }
        }
      }
    }),
    sensitive(yamlencode({
      global = {
        coreServices = {
          license = {
            key = var.license_key
          }
          databases = {
            mongodb = {
              connectionString = var.mongodb_connection_string
            }
          }
          search = {
            embeddingService = {
              apiKey = var.embedding_service_api_key
            }
          }
        }
      }
    }))
  ]

  depends_on = [
    kubernetes_namespace.webdatasource
  ]
}

NOTE This code can be used to create a dedicated Terraform Module for WDS API Server.

Docker Compose Deployment Options

The docker-compose files in an air-gapped environment are a bit different — the docker registry has been changed during loading to a private registry. Therefore, the docker-compose files from this page (with the new docker registry) should be used.

MINI, OEM Business, and BOX Business are the currently reconciled Compose options:

MINI (Free) - runs Solidstack AIO FTVS and the auxiliary containers, including local MongoDB Atlas Search and embedding services.
OEM (Business) - runs the multi-service topology and auxiliary containers while using external database, search, cache, and embedding infrastructure.
BOX (Business) - includes the multi-service topology, MongoDB Atlas Local 8.3.8, cache storage, Ollama, and the embeddinggemma model for vector search.

FIPS: For the OEM and BOX options, replace each webdatasource/<service>:<tag> core service image with its corresponding webdatasource/<service>-fips:<tag> image to run the core services in FIPS-compliant mode. FIPS images are available only for linux/amd64; Playground and Docs are unaffected.

This deployment option uses Solidstack AIO FTVS to bundle MongoDB Atlas Local, Ollama, and the embeddinggemma model in the Solidstack container. Retrieval is enabled in FullTextAndVector mode without external database or embedding services. Its bundled credentials are for evaluation only.


services:
  wds.solidstack:
    image: [registry]/webdatasource/solidstack-aio-ftvs:v3.0.0
    restart: unless-stopped
    hostname: solidstack
    environment:
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - JOB_TYPES=intranet
    ports:
      - 0.0.0.0:2807:8080

  wds.playground:
    image: [registry]/webdatasource/playground:v3.0.0
    restart: unless-stopped
    hostname: playground
    ports:
      - 0.0.0.0:2808:80

  wds.docs:
    image: [registry]/webdatasource/docs:3.0.3
    restart: unless-stopped
    ports:
      - 0.0.0.0:2809:80

This deployment option runs the distributed WDS services and auxiliary containers without third-party infrastructure. Replace each database, search, cache, and embedding placeholder with an external service that is reachable from the Compose network. It includes Dapi, Crawler, Datakeeper, Idealer, Scraper, Retriever, and Jober, plus Playground and Docs.

Replace the value of the EMBEDDING_SERVICE_URL with a URL to an embedding service. For evaluation purposes, this service can be hosted locally:

If it’s a SaaS, the API Key can be set in the EMBEDDING_SERVICE_API_KEY environment variable

By default, system DB (MongoDB) is used to cache web pages. This behavior can be changed by providing the wds.datakeeper service with a CACHE_CONNECTION_STRING.

IMPORTANT! MongoDB connection strings must contain a database name. The names might be the same as values in the example connection strings or others. IMPORTANT! This feature is available starting from the Business Plan


services:
  wds.crawler:
    image: [registry]/webdatasource/crawler:v3.0.0
    restart: unless-stopped
    hostname: crawler
    environment:
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SERVICE_HOST=crawler
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - LICENSE_KEY=[LICENSE]

  wds.dapi:
    image: [registry]/webdatasource/dapi:v3.0.0
    restart: unless-stopped
    hostname: dapi
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Dapi?appName=<cluster>&readPreference=secondary
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - IDEALER_ORIGIN=http://idealer
      - JOBER_ORIGIN=http://jober
      - RETRIEVER_ORIGIN=http://retriever
      - JOB_TYPES=intranet
      - LICENSE_KEY=[LICENSE]
    ports:
      - 0.0.0.0:2807:8080

  wds.jober:
    image: [registry]/webdatasource/jober:v3.0.0
    restart: unless-stopped
    hostname: jober
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Jober?appName=<cluster>&readPreference=secondary
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - DAPI_ORIGIN=http://dapi
      - LICENSE_KEY=[LICENSE]

  wds.datakeeper:
    image: [registry]/webdatasource/datakeeper:v3.0.0
    restart: unless-stopped
    hostname: datakeeper
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Datakeeper?appName=<cluster>&readPreference=secondary
      - CACHE_CONNECTION_STRING=s3://<access-key>:<secret-key>@<host>:<port>/<bucket>?ssl=true
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]

  wds.idealer:
    image: [registry]/webdatasource/idealer:v3.0.0
    restart: unless-stopped
    hostname: idealer
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Idealer?appName=<cluster>&readPreference=secondary
      - LICENSE_KEY=[LICENSE]

  wds.scraper:
    image: [registry]/webdatasource/scraper:v3.0.0
    restart: unless-stopped
    hostname: scraper
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Scraper?appName=<cluster>&readPreference=secondary
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]

  wds.retriever:
    image: [registry]/webdatasource/retriever:v3.0.0
    restart: unless-stopped
    hostname: retriever
    environment:
      - MONGODB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Retriever?appName=<cluster>&readPreference=secondary
      - SEARCH_DB_CONNECTION_STRING=mongodb://<user>:<password>@<host>:<port>/Retriever?authSource=admin&directConnection=true
      - EMBEDDING_SERVICE_URL=http://<host>:<port>/api/embed
      - IDEALER_ORIGIN=http://idealer
      - SEARCH_MODE=FullTextAndVector
      - LICENSE_KEY=[LICENSE]

  wds.playground:
    image: [registry]/webdatasource/playground:v3.0.0
    restart: unless-stopped
    hostname: playground
    ports:
      - 0.0.0.0:2808:80

  wds.docs:
    image: [registry]/webdatasource/docs:3.0.3
    restart: unless-stopped
    ports:
      - 0.0.0.0:2809:80

This deployment option is intended to contain the distributed WDS services, bundled MongoDB/search/cache infrastructure, and auxiliary containers. It runs one instance of each service and uses evaluation-only credentials.

MongoDB Atlas Local supplies the system and Atlas Search databases. A separate Ollama container supplies the embedding service. A one-shot initializer downloads embeddinggemma:latest into the persistent embedding-models volume before Retriever starts, and Retriever calls the internal http://embedding-service:11434/api/embed endpoint.

IMPORTANT! This feature is available starting from the Business Plan


services:
  mongodb:
    image: [registry]/mongodb/mongodb-atlas-local:8.3.8
    restart: always
    hostname: mongodb
    environment:
      - MONGODB_INITDB_ROOT_USERNAME=root
      - MONGODB_INITDB_ROOT_PASSWORD=TestPassword
    volumes:
      - mongodb-data:/data/db
      - mongodb-config:/data/configdb
    healthcheck:
      test: ["CMD-SHELL", "mongosh --host 127.0.0.1 --username root --password TestPassword --authenticationDatabase admin --quiet --eval 'quit(db.adminCommand({ ping: 1 }).ok ? 0 : 2)'"]
      interval: 5s
      timeout: 3s
      retries: 40

  minio:
    image: [registry]/bitnami/minio:2024
    restart: always
    hostname: minio
    environment:
      - MINIO_DEFAULT_BUCKETS=pages-cache
      - MINIO_ROOT_USER=TestAccessKey
      - MINIO_ROOT_PASSWORD=TestSecretKey
      - MINIO_FORCE_NEW_KEYS=no

  embedding-service:
    image: [registry]/ollama/ollama:latest
    restart: always
    hostname: embedding-service
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_MODELS=/root/.ollama/models
    volumes:
      - embedding-models:/root/.ollama
    healthcheck:
      test: ["CMD-SHELL", "OLLAMA_HOST=http://127.0.0.1:11434 ollama show embeddinggemma:latest >/dev/null 2>&1"]
      interval: 5s
      timeout: 3s
      retries: 40

  embedding-model-init:
    image: [registry]/ollama/ollama:latest
    restart: "no"
    depends_on:
      embedding-service:
        condition: service_started
    environment:
      - OLLAMA_HOST=http://embedding-service:11434
    entrypoint:
      - /bin/sh
      - -ec
      - until ollama list >/dev/null 2>&1; do sleep 1; done; ollama show embeddinggemma:latest >/dev/null 2>&1 || ollama pull embeddinggemma:latest
 
  wds.crawler:
    image: [registry]/webdatasource/crawler:v3.0.0
    restart: always
    hostname: crawler
    environment:
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SERVICE_HOST=crawler
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - LICENSE_KEY=[LICENSE]
  
  wds.datakeeper:
    image: [registry]/webdatasource/datakeeper:v3.0.0
    restart: always
    hostname: datakeeper
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Datakeeper?authSource=admin&directConnection=true
      - CACHE_CONNECTION_STRING=s3://TestAccessKey:TestSecretKey@minio:9000/pages-cache?ssl=false
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]

  wds.idealer:
    image: [registry]/webdatasource/idealer:v3.0.0
    restart: always
    hostname: idealer
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Idealer?authSource=admin&directConnection=true
      - LICENSE_KEY=[LICENSE]
  
  wds.scraper:
    image: [registry]/webdatasource/scraper:v3.0.0
    restart: always
    hostname: scraper
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Scraper?authSource=admin&directConnection=true
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]
  
  wds.dapi:
    image: [registry]/webdatasource/dapi:v3.0.0
    restart: always
    hostname: dapi
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Dapi?authSource=admin&directConnection=true
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - IDEALER_ORIGIN=http://idealer
      - JOBER_ORIGIN=http://jober
      - RETRIEVER_ORIGIN=http://retriever
      - JOB_TYPES=intranet
      - LICENSE_KEY=[LICENSE]
    ports:
      - 2807:8080

  wds.jober:
    image: [registry]/webdatasource/jober:v3.0.0
    restart: always
    hostname: jober
    depends_on:
      mongodb:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Jober?authSource=admin&directConnection=true
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - DAPI_ORIGIN=http://dapi
      - LICENSE_KEY=[LICENSE]

  wds.retriever:
    image: [registry]/webdatasource/retriever:v3.0.0
    restart: always
    hostname: retriever
    depends_on:
      mongodb:
        condition: service_healthy
      embedding-model-init:
        condition: service_completed_successfully
      embedding-service:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Retriever?authSource=admin&directConnection=true
      - SEARCH_DB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb:27017/Retriever?authSource=admin&directConnection=true
      - EMBEDDING_SERVICE_URL=http://embedding-service:11434/api/embed
      - IDEALER_ORIGIN=http://idealer
      - SEARCH_MODE=FullTextAndVector
      - LICENSE_KEY=[LICENSE]
  
  wds.playground:
    image: [registry]/webdatasource/playground:v3.0.0
    restart: always
    hostname: playground
    ports:
      - 2808:80
  
  wds.docs:
    image: [registry]/webdatasource/docs:3.0.3
    restart: always
    ports:
      - 2809:80

volumes:
  mongodb-data:
  mongodb-config:
  embedding-models:

Running Docker Compose

After an appropriate deployment option is selected and a docker-compose configuration has been copied to a file (e.g., wds-docker-compose.yml), there are two types of scripts to run docker-compose:

Windows - for Windows OS
Linux - for Linux-based OS including MacOS

NOTE_ If another version is currently running, just replace the old docker-compose configuration with the new one (of the same option) and execute the selected script. The only services with changed versions will be recreated. If the new version is of a different option, execute the docker compose down command beforehand.

set COMPOSE_FILE=wds-docker-compose.yml && docker compose up -d

export COMPOSE_FILE=wds-docker-compose.yml && docker compose up -d

Deploying WDS API Server in Air-Gapped Environments

Script Examples

archive-wds-images.bat

push-wds-images.bat

Download and archive WDS docker images:

Load and push WDS docker images to a private registry:

archive-wds-images.sh

push-wds-images.sh

Download and archive WDS docker images:

Load and push WDS docker images to a private registry:

Deploy to Kubernetes

Deploying using Helm CLI

Deploying using Terraform

SingleService

MultiService

Docker Compose Deployment Options

Running Docker Compose

Please rotate your device to landscape mode