The Browsers We Didn't Need

Browser-based crawling carries a compounding tax: every page costs seconds instead of milliseconds, fleets need orchestration and crash discipline, and every headless instance is a fingerprint surface that can burn an IP pool. The pipeline had also drifted into silent data damage — a field-mapping bug had left hundreds of thousands of records with an empty vehicle-type field, undetected because the dashboard was green. The task was not "make the browser faster"; it was to re-derive the cheapest extraction path from scratch and prove it before touching production.

Probe the transport before assuming the layer. A plain HTTP request returned fully server-rendered HTML — no WAF challenge, no JS gate. Then the discovery that settled it: the same endpoint returns structured JSON when the request simply asks for it. The server renders from that JSON internally; sending the right Accept header gets the data before rendering, skipping HTML parsing entirely on the list path. Detail pages stay HTML-only but are fully server-rendered and cheerio-parseable — still no browser. The recon also exposed the real design constraint: a hard pagination ceiling per query, which is why a sharding pipeline (brand × location × model × price range) exists at all — each shard is sized to fit under the cap. And it caught the silent bug: a vehicle-type field mapped from the wrong source, leaving a six-figure count of records empty. Recon paid for itself twice — once in compute, once in data trust.