Real Estate Agent Scraping: 8 Top New York Brokerages Lead List
Portfolio project
A Python web scraping and lead generation project that built a clean, verified database of real estate agent contacts from 8 of New York’s leading brokerages. I built a custom web scraper for each website, then merged everything into structured Excel files split by market.
Brokerage Websites Scraped
Compass
Corcoran
Douglas Elliman
Sotheby’s International Realty
Brown Harris Stevens
Serhant
Warburg Realty
Nest Seekers International
Markets Covered
New York City / Manhattan
Brooklyn
Long Island
Web Scraping Challenge
The client needed a complete, up-to-date real estate agent lead list for outreach. Every brokerage website is built differently: some use GraphQL APIs, some hidden REST backends, some JavaScript-heavy pages that only load data as you scroll, and some plain HTML. One generic scraper couldn’t handle all of them, so each site needed its own data extraction strategy.
My Web Scraping Approach
GraphQL API scraping: queried the GraphQL endpoints behind Sotheby’s International Realty and Serhant directly for clean JSON agent data.
REST API reverse engineering: found and used Corcoran’s backend agent-search API and its Next.js data endpoints to get full agent profiles.
JSON endpoint scraping: pulled Warburg Realty’s agent search results straight from its internal JSON feed.
Dynamic page scraping with Selenium: automated a browser to handle JavaScript-loaded agent directories and infinite scroll.
HTML scraping with BeautifulSoup: paginated through Compass, Douglas Elliman, Nest Seekers and Brown Harris Stevens directories.
Email enrichment: visited individual agent profile pages to collect missing email addresses.
Data cleaning and normalization: removed duplicates, standardized phone formats and organized the output by brokerage and market.
Extended to Canada: also built a scraper for Realtor.ca’s agent search API.
Scraped Data Delivered
Agent full name
Email address
Mobile and office phone numbers
Office location / address
Agent profile URL
Brokerage and market, for easy segmentation
Lead Generation Results
8 brokerage websites scraped, each with its own custom scraper
3 New York markets: NYC/Manhattan, Brooklyn and Long Island
100% email coverage on every delivered lead
98.9% phone number coverage
Clean Excel files organized by brokerage and market, ready for CRM import and cold outreach

compass realestate agents web scraped
Web Scraping Tech Stack
Python · Web Scraping · GraphQL · REST API · Selenium · BeautifulSoup · Requests · Pandas · Excel · Lead Generation
0 likes · 0 views · Nov 10, 2026
Python web scraping project: built scrapers for 8 top New York real estate brokerages to create a verified agent lead list with 100% email coverage.