Built a distributed data ingestion system for collecting lis...

Built a distributed data ingestion system for collecting listing, lead, and analytics data from multiple real estate data sources. The system used Python, FastAPI, Celery, Redis, Selenium Grid, and Docker to coordinate authenticated scraping across concurrent workers. I designed shared session management using Redis locks and pub/sub, parallelized scraping workloads with Celery, and implemented missing-page reconciliation to improve reliability. The production pipeline was designed to process large ingestion workloads reliably without manual intervention.