End-to-End Data Pipelines.

Developed and deployed data pipelines using Azure Data Factory to ingest, transform, and load complex datasets into the Data Lake (ADLS Gen 2) using REST API.

Implemented real-time monitoring tools to ensure seamless data flow and identify bottlenecks.

Explored ways to extract relevant features from the data to develop an anomaly detection model.

Developed a proof-of-concept solution using Apache Spark and Great Expectations to demonstrate the feasibility of implementing a data validation framework.

Communicated with stakeholders to understand their needs and translated their feedback into actionable insights.

Standardized documentation procedures across projects, facilitating knowledge transfer for future developers.