Facebook Marketplace Scraper Development

This case study details the development of a full-stack website designed to scrape product data from Facebook Marketplace. The system was created to provide a scalable and reliable solution for clients needing to run customized data collection tasks on demand.

Automate Data Collection: To build a system that can efficiently scrape product data from Facebook Marketplace based on user-defined criteria.

Provide User-Friendly Interface: To create a full-stack website where users can easily input search queries and receive results.

Ensure Scalability: To develop a robust architecture that can handle concurrent user requests efficiently.

Automate Notifications: To automatically notify users with the scraped data upon task completion.

A client needed an automated system to scrape product data from Facebook Marketplace. The process of manually searching and collecting this data was time-consuming and inefficient. The client required a solution that would allow users to define specific search criteria and receive the results in a timely manner.

The solution was a full-stack website with an Angular frontend and a Flask backend. A microservices architecture, powered by Celery and Redis, was implemented to handle background tasks for scraping. The scraper, built using a combination of Playwright, Selenium, Requests, and BeautifulSoup, efficiently fetched data based on user input. The collected data was then stored in DynamoDB for high performance and scalability. Users were automatically notified via email once their scraping job was complete.

The project's scope included the development of an end-to-end solution for web scraping. This encompasses the user interface for submitting queries, the backend logic for processing requests, the web scraping mechanism itself, secure data storage, and an automated notification system. The system's architecture was designed to handle a high volume of concurrent user requests while maintaining speed and reliability.

User Interaction: The user submits a search query through the front-end website.

Task Management: The query is passed to a background task runner (Celery) to initiate the scraping job.

Data Scraping: The scraper, using a combination of libraries like Playwright and Selenium, begins scraping all relevant data based on the user's criteria.

Data Storage: The scraped data is saved to a DynamoDB database.

Status Monitoring: A separate process monitors the status of the scraping job.

User Notification: Once the task is complete, an automated email notification is sent to the user with the scraped data or a link to the results.

Frontend: Angular

Backend: Flask (Python)

Libraries/Tools: Playwright, Selenium, Requests, BeautifulSoup

Database: DynamoDB

Cloud Services: AWS

Task Management: Celery, Redis

The Facebook Marketplace Scraper delivered significant improvements in data collection efficiency and user experience:

90% Reduction in Time-to-Data: The system reduced the time to acquire a complete dataset from hours of manual work to just a few minutes, representing a 90% reduction in time spent on data collection.

Over 500% Increase in Efficiency: The automated scraper processed data 5 times faster than manual methods, allowing users to run multiple scraping jobs concurrently.

24/7 System Availability: The system offers continuous support, allowing users to submit queries at any time without manual intervention.

Annual Savings of $50K: By automating the data collection process, the client saved an estimated $50,000 annually in labor costs.

Support for 100+ Concurrent Requests: The robust microservices architecture efficiently handled over 100 concurrent user requests, ensuring a reliable and fast experience.

The Facebook Marketplace Scraper successfully addressed the client's business problem by providing a scalable, reliable, and user-friendly solution for data collection. By automating the scraping process and ensuring a robust architecture, the system empowered users to effortlessly fetch, store, and review data from Facebook Marketplace.