IronFlow: Python ETL Engine for Clean, Trusted Data
Mojtaba Karimiabdar
Six messy rows in, four trusted rows out: types fixed, totals calculated, e-mails masked and every rejected row explained.
The challenge
Most ETL scripts fail the same way: they load half a dataset, leave the database in an unknown state, and quietly drop rows nobody notices.
The solution
IronFlow is an open-source Python ETL engine I built around the opposite guarantees: a failed run changes nothing, and a bad row is never lost or guessed. The terminal on the cover is a real run of the sample pipeline that ships with the project.
A real run, unedited
Six rows of an order export with three typical problems: a duplicate order, a negative quantity and a missing customer ID. One command produced:
4 clean rows, with types fixed, totals calculated and e-mails masked
2 rejected rows, each with the exact rule it broke
A regional revenue summary for reporting
Nothing was guessed and nothing was silently dropped.
How it works: extract from 12+ kinds of sources, clean and reshape, validate every row, then load in one transaction.
How it works
Pipelines are declared in a short YAML file. Each stage streams in small batches, so memory stays flat even on very large files:
Extract: 12+ kinds of sources, including CSV, Excel, JSON, XML, Parquet, PostgreSQL, MySQL, SQLite, REST and GraphQL APIs and SFTP, with incremental loads of only what changed.
Transform: normalize columns, fix types, dates and time zones, join and aggregate, remove duplicates, and mask, hash or encrypt personal data.
Validate: required, unique, range, pattern and e-mail rules; bad rows are quarantined with the reason, and the run stops if too many rows fail.
Load: append, overwrite or upsert inside one transaction, so a failed run changes nothing.
It runs on a schedule, resumes after a failure, sends alerts by e-mail, Slack or webhook, and includes a small dashboard and a REST API.
1,443 automated tests at 91.5% branch coverage, plus PostgreSQL, Docker and security checks on every change.
Quality you can verify
1,443 automated tests at 91.5% branch coverage run on every change, on Linux and Windows and on Python 3.11 to 3.14, together with a real PostgreSQL integration, a Docker image check, CodeQL, a dependency audit and a secret scan.
Tech stack: Python, pandas, SQLAlchemy, Pydantic and Docker.
Source code (MIT licence): github.com/mojtaba-py-code/ironflow
Have messy exports that need to land in a database you can trust? Send me a message with a sample file.
Like this project
Posted Oct 12, 2026
An open-source ETL engine that cleans, validates and loads data from 12+ sources, and never leaves a table half-loaded. 1,443 tests.
