Efficient ETL and Data Ingestion Automation for AnalyticsEfficient ETL and Data Ingestion Automation for Analytics
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Arsen's avatar
Arsen Poghosyan
• Jun 8
Project Name: DeltaPipe
Subtitle: An automated ETL and data ingestion pipeline for validating, processing, and storing analytical data in ClickHouse and Delta formats.
Project Description
DeltaPipe is a lightweight, high-throughput ETL engine designed to automate data ingestion and validation. The system processes incoming binary/raw data, performs runtime validation for structured datasets (like claims and contracts), and safely writes the outputs into optimized formats for analytics.
What I Did:
Automated Data Ingestion: Developed a core ingestion service (analytics.py) that dynamically intercepts raw file uploads (bytes), detects the business entity type (e.g., claims or contracts), and routes them to their respective validation pipelines.
Strict Runtime Validation: Integrated data quality gates (validations.py) to validate schema integrity before any database write occurs, preventing corrupt or malformed data from breaking production tables.
Storage Optimization: Implemented a multi-tier storage mechanism where validated datasets are formatted as localized CSV tracking logs and structured as Delta Lake layers (delta_files/, raw_delta) for optimized historical time-travel analysis.
ClickHouse Ingestion: Co-developed high-performance data loading utilities (etl_clickhouse.py, clickhouse_connector.py) to batch-insert processed records straight into ClickHouse, ensuring fast, real-time query responses for analytical dashboards.
Asynchronous Task Architecture: Organized the backend with a detached worker system (celery_project/) to decouple heavy data-crunching and insertion processes from the primary application logic.
JupyterPythonData EngineeringClickhouse
Post image
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started