Mozambique Economic Data ETL Pipeline by Thiyane XavierMozambique Economic Data ETL Pipeline by Thiyane Xavier

Mozambique Economic Data ETL Pipeline

Thiyane Xavier

Completed work

Backend Engineer

Data Engineer

Software Engineer

Docker

pandas

SQLite

Data

ETL Pipeline: Mozambique Economic Data

Overview

This is an ETL (Extract, Transform, Load) pipeline that processes Mozambique's economic data from the World Bank. The pipeline extracts data from a CSV file, cleans and standardizes it, loads it into a database, and answers business questions about the data.

Pipeline Stages

1. Extract (Bronze Layer)

Reads messy CSV file using pandas

Adds audit columns: _row_hash, _load_timestamp, _load_id

Preserves raw data exactly as received

Output: Parquet files in Data/bronze_raw/

2. Transform (Silver Layer)

Removes rows with missing critical values

Standardizes column formatting (uppercase country codes, title case names)

Replaces inconsistencies in data values

Extracts and standardizes year values from various formats

Removes irrelevant columns

Output: Cleaned Parquet files in Data/silver/

3. Load

Loads cleaned data to SQLite database (database/economy.db)

Stores data in economic_data table for querying

4. Gold Layer (Business Insights)

Answers business questions using SQL queries

BQ1: GDP growth rate year-over-year (1982-2024)

BQ4: Data completeness audit (which indicators have complete data)

Output: Results exported as CSV and Parquet files in Data/gold/

Architecture

The medallion architecture separates data into three states:

Bronze: Raw, untouched data (audit trail)

Silver: Cleaned, standardized data (ready for analysis)

Gold: Business-ready insights and answers

How to Run

Build the Docker Image

Run the Pipeline

This will:

Extract raw CSV → Bronze Parquet files

Clean data → Silver Parquet files

Load to SQLite database

Execute Gold queries

Save results to Data/gold/

Project Structure

Key Features

Data Quality: Row-level hashing for deduplication and integrity tracking Audit Trail: Bronze layer preserves raw data for compliance and debugging Reproducibility: Docker containerization ensures consistent runs across machines Business Insights: Gold layer answers real economic questions Scalability: Medallion architecture scales to multiple data sources