Detection-resistant crawling at web scale

Every anti-bot vendor sells the same promise: we'll tell humans from machines. The truth is messier. Detection is a spectrum, and at scale you're not trying to look human, you're trying to be boring enough that nobody bothers looking.

I lead the crawler infrastructure behind an image-rights platform that finds where clients' images are used across the web. Here's the mental model I keep coming back to.

The fingerprint is the tell

Plenty of crawlers get flagged on the handshake, not the page. TLS parameters, header order, the JS environment: each is a fingerprint, and a mismatch between what your headers claim and what your runtime actually is gets you flagged before you render a single pixel.

// illustrative Puppeteer sketch: one identity per browser, every layer telling the same story
const id = nextIdentity(); // user agent + matching client hints, viewport, proxy exit
const browser = await puppeteer.launch({ args: [`--proxy-server=${id.proxy}`] });
await browser.setCookie(...id.consentCookies); // consent already given: no banner to click
const page = await browser.newPage();
await page.setUserAgent({ userAgent: id.userAgent, userAgentMetadata: id.clientHints });
await page.setViewport(id.viewport);
await page.goto(url, { waitUntil: "networkidle2" });

The win isn't a single clever spoof. It's that the IP, the TLS stack, the headers, and the JS surface all agree with one another. Coherence is the product. On my reverse-image-search crawler, which works against a provider behind layered bot detection, that means a stealth-hardened browser with a dozen hand-written runtime patches, client hints kept coherent with every rotated user agent, consent cookies pre-seeded across regional domains, and randomized viewports.

Scored proxy pools

Datacenter IPs are a tell on their own; a pool of residential exits removes the cheapest signal a WAF has. The hard part isn't acquiring exits. It's scoring them continuously and pulling the bad ones before they sink a job.

My maps crawler rotates residential and premium proxies by an asymmetric health score (EMA-smoothed latency, weighted success rate, +10/−20 on a 0–100 range), so an exit loses standing faster than it earns it. The reverse-image crawler adds cooldown and recovery, so a failing exit sits out instead of being retried into the ground.

Human-shaped pacing

Bots are fast and regular; people are slow and noisy. Between requests, jitter your timings, keep to the rate a real session would produce, and back off the moment a host gets defensive. That takes treating CAPTCHAs and rate limits as their own signals, not generic errors.

Inside a page it can be the opposite: the reverse-image crawler takes a deliberately fast interaction path that races the provider's client-side telemetry. The goal is a rhythm that doesn't stand out, not a slow one.

What still breaks

Nothing here is permanent. It's an arms race, and the other side ships too. The infrastructure that lasts is the one you can observe: Prometheus metrics and per-cell telemetry on the maps crawler, per-image step progress streamed to the reverse-image crawler's dashboard. When something gets blocked, you want to see which step, which exit, and which identity, not just a falling success rate. You don't win detection. You stay boring, and you stay watching.

The reverse-image-search and maps-scraper pages go deeper on both systems. If you need data collection that stays reliable at scale, I'm open to select work.