Built a distributed data ingestion system for collecting listing, lead, and analytics data from m...Built a distributed data ingestion system for collecting listing, lead, and analytics data from m...
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Built a distributed data ingestion system for collecting listing, lead, and analytics data from multiple real estate data sources. The system used Python, FastAPI, Celery, Redis, Selenium Grid, and Docker to coordinate authenticated scraping across concurrent workers. I designed shared session management using Redis locks and pub/sub, parallelized scraping workloads with Celery, and implemented missing-page reconciliation to improve reliability. The production pipeline was designed to process large ingestion workloads reliably without manual intervention.
Post image
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started