Scraper Service
Scraper extracts structured values from downloaded HTML, converts content into supported formats, and stores and retrieves scraped data by identifier.
Release image: webdatasource/scraper:v3.0.0
Configuration
| Name | Required | Default | Meaning |
|---|---|---|---|
MONGODB_CONNECTION_STRING |
Yes | None | MongoDB connection string for scraped data. Include a database name unless MONGODB_DATABASE_NAME is set. |
MONGODB_DATABASE_NAME |
No | Database from the connection string | Overrides the MongoDB database name. |
IDEALER_ORIGIN |
Yes | None | Origin of Idealer; Scraper appends port 8082 for gRPC-Web. |
LICENSE_KEY |
Yes | None | WDS license key. See Plans. |
HEALTH_PROBE_LOG_LEVEL |
No | Debug |
Log level used for /health and /ready request messages. |
MIN_LOG_LEVEL |
No | Info |
Minimum application log level. |
Endpoints and dependencies
Port 8080 serves /health and /ready. Port 8082 serves Scraper gRPC-Web over HTTP/1.1. Scraper uses MongoDB and connects to Idealer over gRPC-Web; Dapi and Jober send it extraction work.