Overview
WDS Server is responsible for crawling through web resource pages and scraping data.
Services
WDS API Server consists of two service types:
- Core Services - the services that handle workload:
-
- Dapi, Datakeeper, Crawler, Scraper, and Idealer - core services stack
-
- Solidstack - all-in-one service for evaluation purposes
- Auxiliary Services - the services that help in testing and evaluation:
Third-Party Components
The following third-party components are required for WDS Server:
- MongoDB - stores all the system’s data.
-
- Supported versions: 6.x, 7.x, 8.x.
-
- Supported deployments: Atlas, Enterprise, Community
Optionally to optimize prices and performance, the following services might be used:
- S3 compatible storage - is used to cache and reuse downloaded web resource pages. If this is not specified, the system MongoDB is used
-
- AWS S3
-
- Minio
-
- Other compatible services. The Minio .NET Client is used to iterate with S3 compatible storages
Depending on the deployment type, third-party components might be provided together with a deployment or are required to be provided.
Deployment Types
Deployment types are not dependent on WDS Server version and might be added to this documentation anytime, so follow the updates.
Currently, deployment types are the following: