Job-market data is abundant, but rarely analysis-ready.

Listings are distributed across multiple platforms, the same opportunity may appear more than once, employment and seniority fields use inconsistent formats, salary units cannot always be compared safely, and closed positions often remain mixed with active opportunities.

I built Global Job Intelligence to transform this fragmented data into a continuously operating market-intelligence product.

The platform collects job listings from six global sources and 60 company boards, processes each source through an independent pipeline, and converts different data structures into a unified analytical model.

V2 now delivers a complete cloud-connected workflow:

• Automated daily ingestion from Himalayas, Jobicy, Remote OK, Arbeitnow, Greenhouse, and Ashby • Monitoring of 30 Greenhouse and 30 Ashby company boards • Source-specific validation, processing, and normalization • PostgreSQL and Neon cloud database integration • Cross-source deduplication using normalized SHA-256 fingerprints • First-seen, last-seen, active, and closed job lifecycle tracking • Daily market snapshots for historical and trend analysis • Salary analysis that keeps incompatible currencies and pay periods separate • Pipeline audit history, source health, execution metrics, and failure isolation • 85 automated tests protecting pipeline, database, analytics, and dashboard behaviour

The interactive Streamlit command center turns the resulting data into five practical views:

• Market Overview for active opportunities, hiring activity, remote share, and data freshness • Market Intelligence for category demand, seniority, companies, and work-mode analysis • Salary Intelligence for responsible compensation comparisons • Job Explorer for search, filtering, source provenance, direct application links, and CSV/JSON export • Data Health for pipeline reliability, source volume, inactive records, and execution history

One of the central design decisions was to preserve source-level records while also identifying a single market opportunity across multiple platforms. This maintains traceability without allowing duplicate listings to distort market metrics.

The result is more than a collection of job listings. It is a structured system for understanding where demand exists, which companies are hiring, how remote opportunities are distributed, how transparent salary data is, and whether the underlying data pipeline remains reliable.

The platform is deployed, connected to a live cloud database, and designed to improve as daily historical snapshots accumulate.

V3 will extend this foundation with AI-powered market analysis, skill extraction, semantic search, job summarization, and candidate-to-job matching.

Live dashboard: https://global-job-intelligence.streamlit.app/ GitHub: https://github.com/niisa0/global-job-intelligence Portfolio: https://niisa0.github.io/

#Python #DataEngineering #PostgreSQL #Streamlit #Plotly #DataAnalytics #Automation #CloudComputing