Demo on books.toscrape.com, a public site built for scraping practice (its prices and ratings are random).
- Polite crawler: 1,000 book pages, at most 1 request per second, retries with backoff, resume from checkpoint, robots.txt checked first. No CAPTCHA or anti-bot bypass.
- Cleaning: "£51.77" to 51.77, "Three" to 3; de-duplicated by UPC; anomaly report.
- SQLite, CSV and Excel exports, change detection between runs, Excel report with charts.
- 21 pytest tests passing. Real run, 24 Oct 2026: 1,000 rows, 0 duplicates, about 17 minutes.
https://github.com/SheldonZhuang/sheldontek-web-scraper-demo
0 likes · 0 views · Oct 26, 2026
Demo: 1,000 books scraped politely from a practice site, cleaned, de-duplicated and exported to SQLite, CSV and Excel. 21 tests pass.
