Automated Web Scraper → Structured Data Pipeline by Ameer HamzaAutomated Web Scraper → Structured Data Pipeline by Ameer Hamza

Automated Web Scraper → Structured Data Pipeline

Portfolio project
Ameer Hamza

The Problem

A B2B sales team needed a consistent pipeline of fresh prospect data — business names, owner contacts, websites, phone numbers, and industry tags — pulled from specific directories and platforms. Buying data lists was expensive and outdated. Manual research was slow and inconsistent. They needed a system that could pull fresh data on demand, at scale, from the exact sources they specified.
The secondary problem: raw scraped data is messy. Inconsistent formats, missing fields, duplicate records, junk entries. Someone was spending hours every week just cleaning spreadsheets before the data was usable.

What I Built

A Python-based scraping system that handles the full data pipeline — extraction, cleaning, deduplication, and delivery — without any manual steps in between.
The scraper uses cloudscraper to handle anti-bot protection on target sites, rotating through request patterns to avoid detection. It extracts the target fields for each business record, normalizes inconsistent formats (phone numbers, addresses, names), runs deduplication logic to remove records already in the database, and validates emails against a basic format and domain check.
Clean records get pushed directly into Airtable through the API — already tagged by industry, region, and source. The team sees a fresh batch of validated prospects waiting for them each morning. A summary report goes out via email showing how many records were added, how many were skipped as duplicates, and any sites that blocked the scraper that day.
I built this to run on a schedule via a cron job — fully unattended. No one needs to start it, monitor it, or clean the output.

The Result

The team went from spending 3–4 hours/week on manual prospecting and data cleaning to reviewing a clean, ready-to-work list each morning. Data quality improved significantly — duplicate rate dropped from ~30% (typical for manual lists) to under 2%. Cost per record dropped to near zero compared to purchased data lists.
Tech used: Python, cloudscraper library, BeautifulSoup (HTML parser), pandas (data cleaning), Airtable API, email validation logic, cron scheduler, automated summary email via SMTP.

0 likes · 0 views · May 2, 2026

Built a scraper that extracts thousands of business records from the web and delivers them clean, structured, and ready to use

Tags