maps-scraper

A Google Maps business-data crawler built around one hard limit: a search returns at most ~120 results. It subdivides any rectangle or freehand polygon into overlapping grid cells, each kept under the cap, using a Turf.js geometry engine with per-cell optimal-zoom selection and polygon-intersection filtering.

It is a distributed system rather than a script. A Fastify API and a separate BullMQ/Valkey worker coordinate through a singleton-worker lock, per-cell distributed locks, and atomic SELECT … FOR UPDATE SKIP LOCKED cell claiming. A self-healing health monitor recovers stalled cells, orphaned jobs, and abandoned locks after browser-pool crashes, with no human intervention.

The scraping layer drives pooled stealth headless Chrome with network-pattern-based waiting and classified CAPTCHA/rate-limit detection. An asymmetric proxy health score (EMA-smoothed latency, weighted success rate, +10/−20 over a 0–100 range) rotates residential and premium proxies by quality.

Results land in a 10-table PostgreSQL schema with earthdistance geo-indexing and tsvector full-text search, evolved through 12 reversible migrations. A React/Ant Design dashboard adds job wizards, live Leaflet grid visualization, and CSV/JSON/XLSX export. It runs as a PM2-managed three-process deployment over Dockerized PostgreSQL and Valkey, with Prometheus metrics and structured per-cell telemetry: roughly 61,000 lines, owned end to end from geometry math to operational tooling.