Jober Service
Jober stores Traversal configurations, starts scheduled or on-demand Traversal runs, coordinates crawl and scrape tasks, recovers abandoned crawl work, and returns Traversal results and data cursors.
Release image: webdatasource/jober:v3.0.0
Configuration
| Name | Required | Default | Meaning |
|---|---|---|---|
MONGODB_CONNECTION_STRING |
Yes | None | MongoDB connection string for Traversal configuration and run state. Include a database name unless MONGODB_DATABASE_NAME is set. |
MONGODB_DATABASE_NAME |
No | Database from the connection string | Overrides the MongoDB database name. |
DAPI_ORIGIN |
Yes | None | Origin of Dapi; Jober appends port 8082 for gRPC-Web callbacks. |
DATAKEEPER_ORIGIN |
Yes | None | Origin of Datakeeper; Jober appends port 8082 for gRPC-Web. |
SCRAPER_ORIGIN |
Yes | None | Origin of Scraper; Jober appends port 8082 for gRPC-Web. |
TASKS_GET_RESULT_RETRY_DELAY_MS |
No | 1000 |
Delay in milliseconds between task-processing iterations and result checks. |
CRAWL_TASK_FAILURE_RECOVERY_DELAY_SEC |
No | 600 |
Age in seconds after which an abandoned captured crawl task can be recovered. |
LICENSE_KEY |
Yes | None | WDS license key. See Plans. |
HEALTH_PROBE_LOG_LEVEL |
No | Debug |
Log level used for /health and /ready request messages. |
MIN_LOG_LEVEL |
No | Info |
Minimum application log level. |
Endpoints and dependencies
Port 8080 serves /health and /ready. Port 8082 serves Jober gRPC-Web over HTTP/1.1. Jober uses MongoDB and connects to Dapi, Datakeeper, and Scraper over gRPC-Web.