Project Overview
Designed and implemented a metadata-driven data ingestion pipeline using Azure Data Factory (ADF) and Azure Data Lake Storage Gen2 (ADLS Gen2) to automate the ingestion of multiple CSV files.
The solution eliminates the need to create individual pipelines for every file by dynamically discovering source files and orchestrating ingestion through parameterized datasets and reusable pipeline components.
Architecture & Implementation
1. Source Data
Multiple CSV files stored in Azure Data Lake Storage Gen2.
Raw source files organized in a structured folder system.
2. Metadata-Driven File Discovery
Implemented the Get Metadata activity.
Dynamically retrieved file lists using the Child Items property.
Enabled automatic discovery of available source files.
3. Pipeline Orchestration
Used ForEach activity to iterate through discovered files.
Implemented dynamic pipeline execution for each file.
Eliminated repetitive pipeline development.
4. Dynamic Data Ingestion
Configured Copy Data activities.
Used dynamic expressions such as
@item().name.Implemented parameterized source and sink datasets.
Preserved original filenames during ingestion.
5. Data Lake Storage
Ingested files into Azure Data Lake Storage Gen2.
Organized ingested data within the Bronze layer.
Maintained a structured and scalable storage architecture.
6. Deployment & Reusability
Used Azure Resource Manager (ARM) templates for infrastructure deployment.
Designed reusable and parameterized pipeline components.
Key Technical Highlights
Metadata-driven ingestion architecture.
Dynamic file discovery and processing.
Parameterized datasets and pipeline execution.
ForEach-based orchestration.
Automated CSV ingestion.
ADLS Gen2 integration.
ARM template-based deployment.
Technology Stack
Azure Data Factory | Azure Data Lake Storage Gen2 | ARM Templates | SQL | ETL/ELT | Dynamic Pipelines
Project Outcome
Built a reusable and scalable data ingestion solution that automates the movement of multiple CSV files into Azure Data Lake Storage Gen2, reducing repetitive pipeline development and establishing a foundation for downstream data engineering workflows.
1 like · 0 views · Nov 18, 2026
Built a dynamic Azure Data Factory pipeline to automate CSV ingestion into ADLS Gen2 using metadata-driven discovery, ForEach, parameterized datasets.