The Gate That Never Fires

Two gates, opposite natures. Gate one inspects the TLS handshake — any HTTP client with a non-browser fingerprint gets rejected before headers are even read. Gate two is a behavioral JS challenge that only executes inside a real browser. The trap: engineers who pass gate one with a browser then fight gate two head-on — headless detection, canvas noise, behavioral scoring — the hardest and most fragile path. Meanwhile the inventory itself was a moving target: the site displayed one national count, its own facet system implied a number roughly 40% higher, and an independent team's pipeline reported a third figure. Nobody knew the true inventory.

Recon before engineering. A two-second TLS probe with a Chrome-fingerprinted HTTP client killed the brief's central premise: the edge gate inspects the handshake, not the headers, and a correct JA3 gets a clean 200 with full server-rendered HTML. That single fact made gate two irrelevant — the behavioral challenge never fires because its JavaScript is never executed. The data was in the SSR payload all along. Coverage was the harder problem. Each search query silently caps its pagination, so full inventory required recursive sharding. A price-only shred diverged — dealers price at round numbers, so a narrow price window can still hold thousands of listings. The fix was discovered by experiment: shred by registration year first (discrete, breaks the round-number density), then price. The site's own facet counts — six independent, mutually-exclusive dimensions each summing to the same raw universe — served as free ground truth, reconciling all three conflicting inventory figures and proving the crawler's coverage mathematically instead of by assumption.