Batch ETL Data Warehouse Modernization Led

Batch ETL Data Warehouse Modernization

Led modernization of a legacy batch ETL pipeline into a scalable AWS-based data warehouse. Replaced manual SQL jobs with an automated system loading diverse sources into Amazon Redshift.

Ingestion: Daily CSVs and API data land in S3; S3 events trigger Lambda to launch Glue Crawlers and update the Data Catalog.

Transformation: Glue PySpark jobs join and cleanse sales, inventory, and customer data, apply quality checks, and move errors to a quarantine path.

Output: Partitioned Parquet files stored in S3 for loading into Redshift.