We have written multiple ETL ("extract, transform, and load") backend services, which were further used for reporting and analytics purposes by other teams!
ETLs run daily, sometimes every few hours; they have heavy data to dump into ETL DB!
Every ETL processes over 100k records on every run.
We had written ETL for a one-time migration of data over 4 TB.
We have written APIs to generate CUSTOM reports and auto-sent them via email.
Like this project
Posted Oct 6, 2024
ETLs services which runs daily on the on extensive data generated by the main services