Real Estate Agent Scraping: 8 Top New York Brokerages Lead List by Muhammad Jahanzaib TahirReal Estate Agent Scraping: 8 Top New York Brokerages Lead List by Muhammad Jahanzaib Tahir

Real Estate Agent Scraping: 8 Top New York Brokerages Lead List

Portfolio project
Muhammad Jahanzaib Tahir
A Python web scraping and lead generation project that built a clean, verified database of real estate agent contacts from 8 of New York’s leading brokerages. I built a custom web scraper for each website, then merged everything into structured Excel files split by market.

Brokerage Websites Scraped

Compass
Corcoran
Douglas Elliman
Sotheby’s International Realty
Brown Harris Stevens
Serhant
Warburg Realty
Nest Seekers International

Markets Covered

New York City / Manhattan
Brooklyn
Long Island

Web Scraping Challenge

The client needed a complete, up-to-date real estate agent lead list for outreach. Every brokerage website is built differently: some use GraphQL APIs, some hidden REST backends, some JavaScript-heavy pages that only load data as you scroll, and some plain HTML. One generic scraper couldn’t handle all of them, so each site needed its own data extraction strategy.

My Web Scraping Approach

GraphQL API scraping: queried the GraphQL endpoints behind Sotheby’s International Realty and Serhant directly for clean JSON agent data.
REST API reverse engineering: found and used Corcoran’s backend agent-search API and its Next.js data endpoints to get full agent profiles.
JSON endpoint scraping: pulled Warburg Realty’s agent search results straight from its internal JSON feed.
Dynamic page scraping with Selenium: automated a browser to handle JavaScript-loaded agent directories and infinite scroll.
HTML scraping with BeautifulSoup: paginated through Compass, Douglas Elliman, Nest Seekers and Brown Harris Stevens directories.
Email enrichment: visited individual agent profile pages to collect missing email addresses.
Data cleaning and normalization: removed duplicates, standardized phone formats and organized the output by brokerage and market.
Extended to Canada: also built a scraper for Realtor.ca’s agent search API.
# GraphQL API scraping: paginated agent search, clean JSON output
for page in range(1, total_pages + 1):
payload = {"query": AGENT_SEARCH_QUERY,
"variables": {"area": "new-york-city-ny", "page": page}}
agents = session.post(GRAPHQL_URL, json=payload).json()["data"]["agents"]

for a in agents:
leads.append({
"name": a["name"],
"mobile": a.get("mobile"),
"office": a.get("office"),
"email": a.get("email"),
"profile": BASE_URL + a["canonicalUrl"],
})

Scraped Data Delivered

Agent full name
Email address
Mobile and office phone numbers
Office location / address
Agent profile URL
Brokerage and market, for easy segmentation

Lead Generation Results

8 brokerage websites scraped, each with its own custom scraper
3 New York markets: NYC/Manhattan, Brooklyn and Long Island
100% email coverage on every delivered lead
98.9% phone number coverage
Clean Excel files organized by brokerage and market, ready for CRM import and cold outreach
compass realestate agents web scraped
compass realestate agents web scraped

Web Scraping Tech Stack

Python · Web Scraping · GraphQL · REST API · Selenium · BeautifulSoup · Requests · Pandas · Excel · Lead Generation

0 likes · 0 views · Nov 10, 2026

Python web scraping project: built scrapers for 8 top New York real estate brokerages to create a verified agent lead list with 100% email coverage.

Tags