Lead Generation Data Extraction by Dev@ Diraiah

Lead Generation Data Extraction

Dev@ Diraiah

Email Marketer

Business Development Specialist

Lead Generator

BeautifulSoup

Python

SQL

Worked with dozens of freelancers, none were able to bypass the Cloudflare security. I am really greatful to have found your service for long term collaboration.

Table of Contents

Client's Objective

Project Specification

Project Timeline

Challenges/Proposed Workarounds

Solution Provided

Database Preview

Project 2 Database (1.4 Million Rows)

Project 1 Database (700,000 Rows)

Client's Objective

I am looking for a data scrapping expert. This is not for beginners. My data source has over a million records with IP detectors that detects the scrappers if a particular IP visits pages more than 10 times, also it has cloudflare protection and Bot detector being invoked after a few visits. The website even limits the number of times humans can visit the data before they're given a cooldown period.

It has 3 levels of securities and almost every scrapper out here and on Fiverr declined the job saying it's not doable. I do not intend on wasting any more money.

Project Specification

Search for businesses going by the alphabetical order in the business listing page

For each alphabet, Around 30,000 business records are available

Retreive all the data for each business, mainly:

Save all the html data in a tabular form in an XLS or a CSV file

Project Timeline

Client's TImeline: Flexible but less than 3 Months if possible

Scripting: 3 Days

Execution: 7 Days

Delivery: 10 Days

Challenges/Proposed Workarounds

Challenge: Upon initially auditing of the website, it was found that a lot of businesses were repeated through out multiple pages which made the script inefficient and slow constantly hitting on links that were already previously targeted.

Workaround: Inspecting the network requests revealed that all the business data were being fetched from an external database as an API call, Instead of the alphabetical approch, it made much more sense to directly call the API with the business id no which was a serialised ID. This helped the scraping speed up by 30x!

Challenge: The website was one of the toughest to beat in terms of security as it possibly hindered every approch, Apart from just IP blocking it also blocked the network and device so no devices connected to the same network was ever able to make a request.

Workaround: A carefully scripted program that used multiple levels of anti detection measures was developed.

Solution Provided

The work was delivered keeping the clients main objective ie Email Marketting and Lead Generation into the main focus:

770,000 Rows were delivered for Project 1 and 1.4 Million Rows for Project 2

All the rows were carefully examined to remove any untidy or useless characters

Data was filtered in a way that only rows containing the most significant columns prescribed by client was packaged for delivery.

The businesses were sorted by RegistrationDate in the final output as requested

Error tracking using logs to minimise any skipping or failure of data

A suite of tools had to be programmed all serving specific needs

relevant scraper files

Database Preview

Project 2 Database (1.4 Million Rows)

Database Project 1

Project 1 Database (700,000 Rows)

Databse for Project 2

Like this project

Bypassing multilayer antiscraping protocols, Extracted vast datasets over 2 million business details for email marketing and provided seamless delivery.

Likes

Views