The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Azure Data Factory (ADF) project designed to build scalable and dynamic data pipelines. The project includes pipelines, datasets, linked services, and dataflows to enable end-to-end data ingestion, transformation, and loading.
Following are the steps done in this project -
Created the storage group in resource group and uploaded the files
Then, created the pipeline to load the data from source to destination.
In creation of pipeline, I used the linked services as connection to load the data. Created linked services as mentioned for CSV Files, Azure tables, github.
Load the data from github, and get metadata for all the files present in container and applied transformations using pipeline activites, and used CDC for incremental loading. And used the storage based triggers, to run the piepline if there is any new data activity.
And finally added the destination to load output and created alerts using Logic Apps to get mail if pipeline fails.
Firstly we load the data, and update CDC accordingly, and if there are any rows in the table, trigger will start and pipeline will run.
My Work Link: https://github.com/siri70931/AzureDataFactory
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Related posts
Onboarding workflow diagram for Resplendent Data, a B2B business intelligence platform
Check it out ⭣
Five retail sources, 1,070,374 records, 96 automated checks. Biggest defect: 47% of sales predate their product's launch, carrying 46% of $6.17B revenue.
Every row fixed, flagged, or quarantined under a versioned rules contract; never deleted. Revenue reconciled to $0.00 unexplained variance, verified by four independent engines (pandas x2, MySQL, Power Query).
Governed by a 23-entry decision log; least-privilege DB access.
I bring this rigor to client data: audit first, document every change, reconcile to zero.
Public services rarely operate in isolation.
One citizen or business journey can depend on multiple services, systems, approvals and handoffs. When one dependency slows down, understanding where the journey is blocked and what it affects downstream becomes a systems problem.
CIVARA / 007.01 Public Service Operations Command explores how service teams could see journeys, dependencies, bottlenecks and service health from one operational view.
Not another government dashboard. An exploration of the human operational layer behind joined-up public services.
Independent concept study by Jemirae, using Ghana as a research context. Not commissioned, endorsed by, or affiliated with the Government of Ghana. Demo data only.
#govtech #enterpriseux #servicedesign #digitalgovernment #jemirae
Showing where a journey gets blocked is the part that matters in a dashboard like this.