Data EngineeringAmr Elhagary
✅ Migrated Hadoop Spark ETL Pipelines from a private cloud to AWS EMR, improving scalability and reducing costs.
✅ Designed and implemented a Medallion Architecture on AWS Datalake using DMS, Glue, and Athena Hudi tables.
✅ Built CI/CD pipelines for Airflow-based ETL workflows using AWS CodeBuild.
✅ Implemented Snowflake Observability by capturing Snowflake metrics to CloudWatch and setting up alerting using AWS Lambda and Terraform.
✅ Built an ETL pipeline to migrate MSSQL Server invoices to NetSuite using Azure Fabric.
✅ Designed a Lakehouse Architecture using Azure OneLake, ensuring data governance compliance.
✅ Developed and managed dashboards using AWS Quicksight and Kibana for real-time analytics.
✅ Integrated Workday SOAP API to AWS S3 Datalake for enterprise data ingestion.
What's included
ETL Pipeline
Implement Batch and Streaming Data Pipeline in AWS, GCP and Azure using Apache Spark and Apache Airflow using Python/Java
Lakehouse Architecture
Build Lakehouse Architecture using Medallion architecture
API Integrations
Build API Integration with clean code and high optimized performance
Dashboard BI
Build Dashboard in Power BI and Quicksight
Starting at$20 /hr
Tags
Kafka
PySpark
Python
Redshift
SQL
Database Engineer
Data Engineer
Service provided by
Amr Elhagary Banha, Egypt
Data EngineeringAmr Elhagary
Starting at$20 /hr
Tags
Kafka
PySpark
Python
Redshift
SQL
Database Engineer
Data Engineer
✅ Migrated Hadoop Spark ETL Pipelines from a private cloud to AWS EMR, improving scalability and reducing costs.
✅ Designed and implemented a Medallion Architecture on AWS Datalake using DMS, Glue, and Athena Hudi tables.
✅ Built CI/CD pipelines for Airflow-based ETL workflows using AWS CodeBuild.
✅ Implemented Snowflake Observability by capturing Snowflake metrics to CloudWatch and setting up alerting using AWS Lambda and Terraform.
✅ Built an ETL pipeline to migrate MSSQL Server invoices to NetSuite using Azure Fabric.
✅ Designed a Lakehouse Architecture using Azure OneLake, ensuring data governance compliance.
✅ Developed and managed dashboards using AWS Quicksight and Kibana for real-time analytics.
✅ Integrated Workday SOAP API to AWS S3 Datalake for enterprise data ingestion.
What's included
ETL Pipeline
Implement Batch and Streaming Data Pipeline in AWS, GCP and Azure using Apache Spark and Apache Airflow using Python/Java
Lakehouse Architecture
Build Lakehouse Architecture using Medallion architecture
API Integrations
Build API Integration with clean code and high optimized performance
Dashboard BI
Build Dashboard in Power BI and Quicksight
$20 /hr