Metadata-Driven ETL Pipeline Using Azure Data Factory by Sagar SoniMetadata-Driven ETL Pipeline Using Azure Data Factory by Sagar Soni

Metadata-Driven ETL Pipeline Using Azure Data Factory

Portfolio project
Sagar Soni

Project Overview

Designed and implemented a metadata-driven data ingestion pipeline using Azure Data Factory (ADF) and Azure Data Lake Storage Gen2 (ADLS Gen2) to automate the ingestion of multiple CSV files.
The solution eliminates the need to create individual pipelines for every file by dynamically discovering source files and orchestrating ingestion through parameterized datasets and reusable pipeline components.

Architecture & Implementation

1. Source Data

Multiple CSV files stored in Azure Data Lake Storage Gen2.
Raw source files organized in a structured folder system.

2. Metadata-Driven File Discovery

Implemented the Get Metadata activity.
Dynamically retrieved file lists using the Child Items property.
Enabled automatic discovery of available source files.

3. Pipeline Orchestration

Used ForEach activity to iterate through discovered files.
Implemented dynamic pipeline execution for each file.
Eliminated repetitive pipeline development.

4. Dynamic Data Ingestion

Configured Copy Data activities.
Used dynamic expressions such as @item().name.
Implemented parameterized source and sink datasets.
Preserved original filenames during ingestion.

5. Data Lake Storage

Ingested files into Azure Data Lake Storage Gen2.
Organized ingested data within the Bronze layer.
Maintained a structured and scalable storage architecture.

6. Deployment & Reusability

Used Azure Resource Manager (ARM) templates for infrastructure deployment.
Designed reusable and parameterized pipeline components.

Key Technical Highlights

Metadata-driven ingestion architecture.
Dynamic file discovery and processing.
Parameterized datasets and pipeline execution.
ForEach-based orchestration.
Automated CSV ingestion.
ADLS Gen2 integration.
ARM template-based deployment.

Technology Stack

Azure Data Factory | Azure Data Lake Storage Gen2 | ARM Templates | SQL | ETL/ELT | Dynamic Pipelines

Project Outcome

Built a reusable and scalable data ingestion solution that automates the movement of multiple CSV files into Azure Data Lake Storage Gen2, reducing repetitive pipeline development and establishing a foundation for downstream data engineering workflows.

1 like · 0 views · Nov 18, 2026

Built a dynamic Azure Data Factory pipeline to automate CSV ingestion into ADLS Gen2 using metadata-driven discovery, ForEach, parameterized datasets.

Tags