reverse-image-search

A backend crawler that runs reverse image search at scale against a major provider, built for image-rights enforcement: finding where images are used across the web. The provider sits behind layered bot detection, so anti-detection is a first-class subsystem.

It pairs a stealth-hardened headless-browser fork with a stealth plugin and a dozen hand-written runtime patches. Client-hint data stays coherent with rotated user agents, consent cookies are pre-seeded across many regional domains, viewports are randomized, and a deliberately fast interaction path races client-side telemetry.

Results come back two ways: primarily by intercepting the provider's internal batch RPC (stripping its security prefix and walking the protobuf-as-JSON payloads), with DOM scraping as the fallback, then merged. Everything then passes through a three-layer deduplication pipeline spanning an in-memory set, a relational store, and a document store.

The runtime splits into a Fastify REST/WebSocket API and a queue worker. The worker is guarded by an atomic distributed lock with a heartbeat, Lua-atomic refresh/release, and stale-lock stealing. A browser-cluster pool gives every image its own isolated browser, with stall detection and automatic restart.

A self-healing proxy layer adds weighted multi-factor health scoring with cooldown/recovery and live rotation strategies. A React/Ant Design dashboard streams per-image step progress over WebSocket. It's strict TypeScript end to end (~7,500 LOC across 39 backend modules), and a documented six-round internal audit resolved roughly 130 issues.