Datakeeper Service
Datakeeper stores job settings and downloaded content, manages Crawler registrations and download tasks, and reuses cached pages when their HTTP validators permit it.
Release image: webdatasource/datakeeper:v3.0.0
Configuration
| Name | Required | Default | Meaning |
|---|---|---|---|
MONGODB_CONNECTION_STRING |
Yes | None | MongoDB connection string for Datakeeper state. Include a database name unless MONGODB_DATABASE_NAME is set. |
MONGODB_DATABASE_NAME |
No | Database from the connection string | Overrides the MongoDB database name. |
CACHE_CONNECTION_STRING |
No | Primary MongoDB connection | Selects a separate MongoDB or S3-compatible content cache. See Caching. |
CACHE_DATABASE_NAME |
No | Database from the cache connection string | Overrides the cache database name for database-backed caches. |
IDEALER_ORIGIN |
Yes | None | Origin of Idealer; Datakeeper appends port 8082 for gRPC-Web. |
LICENSE_KEY |
Yes | None | WDS license key. See Plans. |
HEALTH_PROBE_LOG_LEVEL |
No | Debug |
Log level used for /health and /ready request messages. |
MIN_LOG_LEVEL |
No | Info |
Minimum application log level. |
Endpoints and dependencies
Port 8080 serves /health and /ready. Port 8082 serves the main Datakeeper gRPC-Web API over HTTP/1.1, and port 8083 serves Datakeeper Resource Manager gRPC-Web over HTTP/1.1. Datakeeper uses MongoDB, connects to Idealer, and optionally connects to a separate cache provider.
Caching
Downloaded pages are cached so later scraping can reuse their content without repeating unnecessary requests. When a resource supplies ETag or Last-Modified, Datakeeper preserves the validators so Crawler can make conditional requests and reuse still-current content.
Without CACHE_CONNECTION_STRING, Datakeeper stores cached pages in its primary MongoDB database. The supported external cache forms are:
- MongoDB:
mongodb://<username>:<password>@<host>:27017/<database-name>ormongodb+srv://<username>:<password>@<cluster>.mongodb.net/<database-name>. - S3-compatible storage:
s3://<key>:<secret>@<host>:<port>/<bucket>?ssl=true|false.