Distributed Web Crawler A concurrent distributed web crawler built in Go for processing multiple ...Distributed Web Crawler A concurrent distributed web crawler built in Go for processing multiple ...
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Distributed Web Crawler
A concurrent distributed web crawler built in Go for processing multiple URLs in parallel.
The crawler uses multiple workers to fetch and process pages concurrently while controlling request rates and coordinating discovered URLs through Kafka.
Tech stack: Go, Kafka, goroutines, channels, context, HTML parser, Docker
The project demonstrates practical use of Go concurrency patterns, distributed task processing, synchronization, rate limiting, and graceful application shutdown.
Post image
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started