Premier League ETL Pipeline for Top Scorers
Thiyane Xavier
Premier League ETL Pipeline
An end-to-end data engineering pipeline that extracts Premier League top scorer data from the API-Football API, processes it through a medallion architecture, loads it into a local analytical warehouse, and visualises insights in Power BI.
Architecture
Tech Stack
Layer Tool Data Source API-Football v3 Orchestration Python Storage Apache Parquet Warehouse DuckDB Containerisation Docker Visualisation Power BI
Project Structure
Pipeline Logic
Extract
Requests top scorer data from the Premier League 2024/25 season via API-Football
Checks HTTP status code before processing
Flattens nested JSON response into a tabular structure
Saves raw data as Parquet to
Data/Bronze/ with a timestamp load IDTransform
Reads the latest Parquet file from
Data/Bronze/Renames columns for clarity (
id → player_id)Casts data types to integers where appropriate
Drops duplicate players
Adds a
season columnSaves cleaned data to
Data/Silver/Load
Reads from
Data/Silver/Loads data into a DuckDB analytical warehouse
Runs 4 business queries against the warehouse
Business Questions Answered
# Question BQ1 Which player scored the most goals? BQ2 Which player had the most assists? BQ3 Which player has the best average minutes per goal? BQ4 Which player has the best average minutes per assist?
Key Findings (2024/25 Season)
Most Goals — Mohamed Salah (29 goals)
Most Assists — Mohamed Salah (18 assists)
Best mins/goal — Antoine Semenyo (most efficient scorer)
Best mins/assist — Jean-Philippe Mateta (most efficient creator)
How to Run
Prerequisites
Python 3.12+
API key from api-football.com
Setup
Run the pipeline
Run with Docker
Environment Variables
Create a
.env file based on .env.example:Notes
This pipeline uses DuckDB as a local analytical warehouse, equivalent in query pattern to Google BigQuery (the intended production target)
The free tier of API-Football allows 100 requests/day, sufficient for this pipeline which runs in a single call
Power BI connects to the exported CSV from DuckDB for visualisation
Dashboard Preview
4-page Power BI dashboard covering all business questions:
Top scorers by goals
Top scorers by assists
Most efficient scorers (mins per goal)
Most efficient creators (mins per assist)
Like this project
Posted May 26, 2026
Python ELT pipeline extracting football statistics from an external API, structured via SQLite database architectures and automated locally.
