A scraper that "works on my machine" but 403s in production? Check your User-Agent before anythin...A scraper that "works on my machine" but 403s in production? Check your User-Agent before anythin...
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
A scraper that "works on my machine" but 403s in production? Check your User-Agent before anything else.
Python's default urllib User-Agent gets blocked by a huge number of sites instantly. The fix is one line — send a real browser User-Agent — but the failure looks like a hundred other things (auth, rate-limits, IP bans), so people chase the wrong bug for an hour.
The broader lesson from building scrapers daily: the code is easy, the resilience is the job — pagination, rate-limits, anti-bot headers, retries. That's where the value is.
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started