Crawler Service
Crawler downloads pages for running jobs. It executes HTTP requests, applies proxy, cookie, HTTPS, delay, and throttling settings, and registers its usable addresses with Datakeeper Resource Manager.
Release image: webdatasource/crawler:v3.0.0
Configuration
| Name | Required | Default | Meaning |
|---|---|---|---|
DATAKEEPER_ORIGIN |
Yes | None | Origin of Datakeeper; Crawler appends port 8083 for the Resource Manager gRPC-Web endpoint. |
GRPC_DNS_RESOLVER_REFRESH_INTERVAL_SEC |
No | 15 |
DNS resolver refresh interval in seconds for the Datakeeper gRPC-Web connection. |
SERVICE_HOST |
Yes | None | Host advertised for this Crawler. It is combined with http:// and port 8082 to form the registered gRPC-Web URL. |
EXTERNAL_IP_ADDRESS_CONFIGS |
No | None | Comma-separated external-address configurations. Supply at least one usable entry when this Crawler should accept intranet or Internet work. |
MAX_INACTIVE_SEC_TO_REREGISTRAR |
No | 60 |
Inactive period in seconds before Crawler registers itself again with Datakeeper Resource Manager. |
LICENSE_KEY |
Yes | None | WDS license key. See Plans. |
HEALTH_PROBE_LOG_LEVEL |
No | Debug |
Log level used for /health and /ready request messages. |
MIN_LOG_LEVEL |
No | Info |
Minimum application log level. |
Endpoints and dependencies
Port 8080 serves /health and /ready. Port 8082 serves Crawler gRPC-Web over HTTP/1.1. Crawler connects to Datakeeper Resource Manager on port 8083 and periodically registers its own port-8082 address.
External IP getter services
For Internet jobs, requests can be sent through proxies or from registered Crawler addresses. The supported configuration values are:
amazon: resolve the public address throughhttps://checkip.amazonaws.com.intranet: register the Crawler for intranet jobs.- A literal IP address, such as
20.21.22.23: register that address directly.
Examples
Intranet jobs only:
EXTERNAL_IP_ADDRESS_CONFIGS=intranet
Intranet and Internet jobs with automatic public-address discovery:
EXTERNAL_IP_ADDRESS_CONFIGS=intranet,amazon
Internet jobs with a fixed public address:
EXTERNAL_IP_ADDRESS_CONFIGS=20.21.22.23