Production metrics
3.3 min
1,000 URLs end to end
~300
URLs per minute
10/sec
Distributed rate limit
6 wks
From zero to production
Bulk URL Health Checker
Processes 1,000 URLs end to end in 3.3 minutes (~300 URLs/min): submit, enqueue, fetch, classify, extract the title, write to the database, and close the batch. A Redis-coordinated BullMQ queue uses a distributed rate limiter (10 req/sec) and a semaphore (5 concurrent) so multiple workers stay within a safe load. Live progress streams over SSE with a full-state resync on reconnect, PostgreSQL as the source of truth, and shared Zod contracts across a 3-app npm workspace. Learned Fastify and BullMQ from zero and shipped the queue system to production in 6 weeks.
Tech Stack

