MobiMiner - Real Estate Web Scraping System

MobiMiner is a distributed web scraping system designed to extract property listings from real estate websites at scale. The architecture uses Redis-based message queues to coordinate multiple workers and parsers, enabling efficient parallel processing of both listing pages and individual property details.

The system employs a two-stage pipeline: home workers fetch property listing pages while detail workers retrieve individual property information. Dedicated parsers extract structured data from the HTML and persist it to the database. This decoupled architecture allows horizontal scaling by deploying multiple parser instances to handle high-volume scraping workloads.