Large-Scale Data Crawling Infrastructure

Each target market runs a different platform with different HTML structure, different anti-bot vendors, and different data schemas. The system needed to: run campaigns concurrently across markets, handle anti-bot defenses per-platform, normalize heterogeneous data into a unified schema, recover from failures without data loss, and be operable by non-engineers through a management interface.

Layered architecture across four planes: Orchestration Layer: Campaign state machine (PENDING → RUNNING → PAUSED → COMPLETE → ERROR) managed via PostgreSQL with advisory locks preventing duplicate crawl workers. AWS SQS queues decouple discovery from detail collection. Worker Layer: Dockerized Node.js workers deployed on AWS ECS. Each worker pulls tasks from SQS, executes requests through the proxy layer, and writes raw HTML + parsed data back to S3 and PostgreSQL. Workers scale horizontally via ECS task count. Anti-Detection Layer: Per-platform proxy pools (residential/datacenter/Tor), TLS fingerprint selection, request timing with Gaussian jitter, UA rotation, and session-consistent cookie management. Proxy health tracked per-domain with automatic failover. Data Layer: Raw HTML archived to S3. Parsed records normalized into a unified schema with EUR currency conversion and deduplication by canonical identifier. Strapi CMS as the admin interface for campaign configuration and data review. Publication pipeline outputs structured datasets for downstream consumers.