Twitter/X Scraper: 30K+ Profiles Daily (Client Project)
Tran Tien Van
Client project for a social intelligence client (client unnamed): daily follower counts for 30,000 Twitter/X profiles, on a platform that suspends accounts, raises CAPTCHA walls and rate-limits automated reading.
What we built
A sharded Python collector that spreads profile links across pre-authenticated accounts in parallel processes
Saved sessions reused between runs; each follower count read from the profile's JSON-LD
Every run consolidated into one snapshot, with 43 resumable shards
Results
30,000+ profiles tracked per daily run across parallel processes
43 resumable shards; 39,168 distinct profiles in the delivered files
Auto-recovery from suspensions, rate limits and CAPTCHA walls
What the data says
Gini coefficient 0.993: the top 1% of accounts hold 96% of all followers
Only 51% of profiles resolved; the rest were deleted, suspended or private
Plan reach on the median account, not the mean
The dashboard was built by Van Data Team for its portfolio from the delivered data; the client engagement delivered the data.
Stack: Python, multiprocessing, browser automation, JSON-LD parsing.
Full case study: vandatateam.com/portfolio
Like this project
Posted Nov 5, 2026
Client project: sharded Python collector tracks 30,000+ Twitter/X profiles per daily run, with session reuse and recovery from rate limits and CAPTCHAs.
Likes
0
Views
0
