Data Engineering project using Apache Airflow and AWS
Nabeel Farooq
As an experienced Big Data Engineer having expertise in Airflow, I will orchestrate seamless data workflows. With expertise in DAG design and workflow automation, I will optimize data pipelines for efficiency and reliability.
I am proficient in Python scripting and task scheduling, I will ensure data integrity and timely execution. I have experience in integrating diverse data sources and implementing error-handling mechanisms, my solutions guarantee robust ETL processes.
Leveraging Airflow's features, I bring order to complex data ecosystems. From custom operators to DAG optimization, my portfolio showcases a track record of streamlining data processes for enhanced business intelligence. Let's elevate your data architecture together!
I will install Apache Airflow, and will create end-to-end python based data pipelines and will orchestrate using Apache Airflow. I will develop two separate DAGs (Directed Acyclic Graphs). The first DAG focuses on fetching data from an API/source and the second DAG will be triggered to perform data processing.
At the end, I will provide a complete detailed documentation of the solution!
Like this project
Posted Jul 17, 2024
Developed an end to end python based ETL Pipeline, orchestrated using Apache Airflow. Developed separate DAGs to fetch and process data and query using Athena.
