Let'sGetChecked Cloud Migration Project

I led the migration of Let'sGetChecked's data system from on-premises to the cloud, maintaining minimal changes to the existing architecture. The process involved:

Data Extraction: Utilizing Airflow DAGs for incremental data loading from SQL Server to S3.

Data Transformation: Using Airflow to trigger Spark/Databricks jobs for processing raw data in S3 and storing processed data back in S3.

Data Loading: Airflow managed the loading of both processed and unprocessed data from S3 into Redshift.

Data Aggregation and Transformation: Further transformations and aggregations were conducted in Redshift using Airflow.

Analytics and Reporting: Data in S3 was accessed via Athena for ad-hoc analytics, while Power BI was used for visualization from Redshift.

This migration ensured a scalable, cloud-based data infrastructure that supports advanced analytics and reporting, improving overall performance and data integrity.