Let'sGetChecked Cloud Migration Project
Darshan Singh
I led the migration of Let'sGetChecked's data system from on-premises to the cloud, maintaining minimal changes to the existing architecture. The process involved:
Data Extraction: Utilizing Airflow DAGs for incremental data loading from SQL Server to S3.
Data Transformation: Using Airflow to trigger Spark/Databricks jobs for processing raw data in S3 and storing processed data back in S3.
Data Loading: Airflow managed the loading of both processed and unprocessed data from S3 into Redshift.
Data Aggregation and Transformation: Further transformations and aggregations were conducted in Redshift using Airflow.
Analytics and Reporting: Data in S3 was accessed via Athena for ad-hoc analytics, while Power BI was used for visualization from Redshift.
This migration ensured a scalable, cloud-based data infrastructure that supports advanced analytics and reporting, improving overall performance and data integrity.
Like this project
Posted Jul 25, 2024
I migrated Let'sGetChecked's data system from on-premises to the cloud, using Airflow,Spark/Databricks,and Redshift for performance and scalability enhancement.
