I build API ingestion pipelines that survive when the downstream fails. Most pipelines I audit lo...I build API ingestion pipelines that survive when the downstream fails. Most pipelines I audit lo...
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Peter's avatar
Peter Aboderin
• Aug 29
I build API ingestion pipelines that survive when the downstream fails.
Most pipelines I audit look fine in the demo and die in production. A 500 error from a third-party API at 2 AM shouldn't mean your entire batch job disappears with no trace. But that's exactly what happens when your system has no retry logic, no circuit breaker, and no dead letter queue.
This pipeline handles it differently. POST a list of URLs, get a job ID back immediately. Processing runs in the background — your API never blocks. If a downstream returns 429 or 500, it retries 3 times with jittered exponential backoff. If a domain fails 5 times in 2 minutes, the circuit breaker opens and stops the bleeding. Every failure is stored with the URL, error, and status code. Nothing gets lost.
Tested live against httpbin.org: working endpoints return 200 in ~1s with zero retries. Failing endpoints get 3 retries, then surrender gracefully with full error capture.
FastAPI + httpx + SQLite. Dockerized. CI-tested across Python 3.10–3.13.
Typical work: e-commerce price monitoring, DeFi/CEX price feeds, news aggregation, any data flow that needs to survive unreliable APIs.
Send me one unreliable API endpoint and what you need extracted. I'll return a sample pipeline architecture in 24 hours — no cost, no commitment.
Completed work
Backend DevelopmentWeb ScrapingSoftware DevelopmentDockerGitHubPython
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started