Relay: a distributed job queue on Redis Streams

The problem: almost every product needs background jobs, for emails, reports, imports and AI calls. The hard part is what happens when a worker dies halfway through a job. Does the job run twice, never, or get stuck forever?

What I built: a distributed job queue in Python on Redis Streams and PostgreSQL. It supports priorities, delayed and recurring jobs, retries with backoff, a dead-letter queue with replay, idempotency keys, cancellation, worker heartbeats, and locks with fencing tokens so a slow worker that wakes up late cannot overwrite newer work.

How I tested it: besides unit and integration tests on real Redis and Postgres, I ran load tests and chaos tests that kill workers on purpose while jobs are running, then check that nothing was lost. I also tested against network faults.

Measured on a busy laptop: about 1,270 jobs per second drained with 8 worker processes, and a p50 of 12 ms from submit to complete at 100 jobs per second. The load test showed the single-job producer falling behind at 500 jobs per second, so I added batching, which then kept up at 1,000 per second.

Details: Python, Redis Streams, PostgreSQL, FastAPI, a small dashboard, Prometheus metrics, k6. 68 tests. It has not run in production and the numbers come with caveats listed in the repository. This is a personal project. Code is on my GitHub: chaitanya-mhetre, repository distributed-job-platform.