Idealer Service

Idealer allocates and persists stable identifiers used by the other WDS services and supports tenant, job, and task-scoped cleanup requests.

Release images:

Note: The regular service image is multi-platform. The -fips service image is built only for linux/amd64.

Configuration

Name Required Default Meaning
MONGODB_CONNECTION_STRING Yes None MongoDB connection string for identifier state. Include a database name unless MONGODB_DATABASE_NAME is set.
MONGODB_DATABASE_NAME No Database from the connection string Overrides the MongoDB database name.
LICENSE_KEY Yes None WDS license key. See Plans.
HEALTH_PROBE_LOG_LEVEL No Debug Log level used for /health and /ready request messages.
SOURCE_HASH_MODE No FullLength Source encoding: FullLength (SHAKE256 with source.Length output bytes), Sha512 (64-byte SHA-512), or Raw (unchanged text). Hashes are Base64-encoded.
MIN_LOG_LEVEL No Info Minimum application log level.

Identifier source hashing

FullLength is the backward-compatible mode. It hashes the UTF-8 source with SHAKE256 and requests source.Length output bytes.

Sha512 produces 64 hash bytes, stored as an 88-character Base64 string.

Raw stores the original source text unchanged. It is mainly intended for testing and evaluation and is not recommended for production.

Select the mode before populating identifier state: changing it on an existing database changes lookup keys and can allocate different IDs for the same source. There is no automatic key migration.

Endpoints and dependencies

Port 8080 serves /health and /ready. Port 8082 serves Idealer gRPC-Web over HTTP/1.1. Idealer’s runtime dependency is MongoDB; Datakeeper, Scraper, and Retriever call it over gRPC-Web.

Please rotate your device to landscape mode

This documentation is specifically designed with a wider layout to provide a better reading experience for code examples, tables, and diagrams.
Rotating your device horizontally ensures you can see everything clearly without excessive scrolling or resizing.

Return to Web Data Source Home