News Scraping & Aggregation: One Clean, Tagged Feed

Teams that follow the news check many sites every day: BBC, The Guardian, TechCrunch, security blogs. The same story shows up in several feeds, nothing is tagged, and it is hard to see what is trending.

I built a news aggregation platform in Python that collects articles from 11 live sources into one database, removes duplicates, tags every story by topic and sentiment, and serves it through a REST API and a live dashboard.

251 real articles from 11 sources, collected in under 30 seconds.

Article volume by hour, sentiment (104 neutral, 41 negative, 27 positive) and topic share (Technology, World, AI, Cybersecurity and more).

Trends across sources: Technology was covered by 10 of the 11 sources at once.

Collect: RSS, REST APIs, JSON Feed or a polite scraper. A new source is one entry in a config file.

Fetch safely: robots.txt and delays respected, private addresses refused, every redirect checked.

Clean: HTML, dates and links tidied up, tracking parameters removed.

De-duplicate in 4 levels: URL, content fingerprint, SimHash and TF-IDF similarity. On the first run, 3 stories that appeared in two feeds were stored once; on the second run, 248 of 251 items were skipped and only the 3 new ones were stored.

Enrich: language, sentiment, keywords, people and organisations, topics and a short summary for every article.

Serve: 62 REST endpoints with search and filters, a live dashboard, and keyword alerts by webhook.

URL safety policy: a source pointing at the cloud metadata address or localhost is refused (422).

Accounts: Argon2id passwords, JWTs with a pinned algorithm, logout that revokes tokens, and user, analyst and admin roles; a wrong password gets the same reply as an unknown user.

Hostile feeds handled: XML parsed with defusedxml, size caps on every response, HTML escaped and a strict Content Security Policy.

Tested on live data: a real run against the 11 feeds found 4 bugs the tests had missed (a startup crash with the sample settings file, a lost batch when two feeds shared a story, an article endpoint error and blocked dashboard charts). Each one is fixed with a test that fails without the fix.

488 automated tests at 85% coverage on SQLite and PostgreSQL, plus ruff, strict mypy, bandit, pip-audit and a Docker smoke test in CI.

Tech stack: Python, FastAPI, async SQLAlchemy, PostgreSQL, Redis, httpx, defusedxml, Typer and Docker.

Source code: github.com/mojtaba-py-code/news-intelligence-platform

Need news, blogs or industry sources collected into one clean feed? Send me a message with the sources and topics you follow.