Job Data Scraping & Analytics: Live Job Market Data

Recruiters, HR-tech products and training companies want to know which skills employers really ask for, where the jobs are and how senior they are. Job postings are spread across hundreds of career sites, full of duplicates and messy text.

I built a job market data platform in Python that collects real postings from public job-board APIs, cleans them, removes duplicates, extracts the skills from every posting and turns the result into market analytics, a REST API and a live dashboard.

2,149 real postings from GitLab, Cloudflare, Datadog, Airbnb, Figma and Discord (Greenhouse), Spotify and Palantir (Lever) and the Arbeitnow API.

1,519 unique jobs from 135 companies in 30+ countries, with 0 errors.

The dashboard shows postings over time, seniority (senior roles lead) and countries (the US leads; Canada is 86% remote).

Collect: Greenhouse and Lever career boards, job APIs, RSS and datasets. A new source is one line of config, and robots.txt is always respected.

Validate: bad records are never dropped silently. In the real run, 119 postings older than 2 years and 16 without a description were quarantined with the exact reason.

Normalize: titles, locations, companies and pay are cleaned up, each with a confidence score.

De-duplicate in 4 stages: source ID, URL, content fingerprint and MinHash. 478 duplicates were removed, mostly the same role posted in several cities.

Extract skills: 123 skills, with required and nice-to-have kept apart. Python appears in 244 postings (19.3%), most often together with Bash, Terraform and Docker.

Analyze: skill demand, trends, co-occurrence, countries, remote share and a market-fit score for a candidate profile.

Salaries are never invented: these APIs have no pay field, so salary statistics stay empty instead of guessing.

SSRF guard: every URL is checked and DNS-resolved before a connection opens. The cloud metadata address, localhost, plain http and file paths were all refused in the real run.

Scoped API keys: keys are stored hashed and carry scopes; admin routes answer 401 without a valid key.

Hostile input handled: XML parsed safely (no XXE), paths can't escape their folder, and logs redact secrets.

Tested on live data: the real run found 4 bugs the tests had missed (Greenhouse jobs lost their description, HTML tags left in job text, a stray letter "c" counted as C++, and multi-word search found nothing). Each one is fixed with a test that fails without the fix.

411 automated tests at 84% coverage, plus ruff, mypy type checks, bandit, pip-audit, PostgreSQL migrations and a Docker smoke test in CI.

Tech stack: Python, FastAPI, SQLAlchemy, PostgreSQL, DuckDB, Parquet, Redis Streams and Docker.

Source code: github.com/mojtaba-py-code/real-time-job-market-intelligence

Need job postings or other public listings turned into clean, searchable data? Send me a message with the sources you have in mind.