Arsen Poghosyan
Python Developer | Data & AI Automation Engineer
New to Contra
Arsen is building their profile!
Subtitle: A dynamic data transformation engine and preview processor designed to parse, map, and process custom metadata schemas at runtime.
Project Description
ModelBuilder is an advanced data modeling backend service built to handle complex, dynamic data transformations. Its core feature is a flexible preview processing engine (preview_processing) that allows users to map, rename, and transform datasets on the fly based on custom payloads and dynamic schemas before committing them to a database.
What I Did:
Dynamic Column Transformation: Developed the runtime mapping logic (processors.py (http://processors.py)) that extracts and transforms column metadata. The engine dynamically checks strategies like manage, create, or rename to dynamically extract column names from nested dictionary payload structures (col_info['used_cols']).
Metadata-Driven Processing Layer: Built a decoupled processing mechanism (process_columns) that extracts custom values, identifies the required data process type, and instantiates factory classes (processor_class) to process raw data, metadata, and error handling configurations dynamically.
Factory & Modular Architecture: Structured the microservice under a clean, modular layout with factory functions (factory_functions), isolated database models (models/), custom middlewares, and dedicated migration scripts.
Production Deployment Configuration: Set up targeted Docker configurations with both app.Dockerfile (for development) and prod.app (http://prod.app).Dockerfile (optimized for production usage), coupled with helper shell scripts (entrypoint.sh (http://entrypoint.sh), run.sh (http://run.sh)).
0
10
Project Name: DeltaPipe
Subtitle: An automated ETL and data ingestion pipeline for validating, processing, and storing analytical data in ClickHouse and Delta formats.
Project Description
DeltaPipe is a lightweight, high-throughput ETL engine designed to automate data ingestion and validation. The system processes incoming binary/raw data, performs runtime validation for structured datasets (like claims and contracts), and safely writes the outputs into optimized formats for analytics.
What I Did:
Automated Data Ingestion: Developed a core ingestion service (analytics.py (http://analytics.py)) that dynamically intercepts raw file uploads (bytes), detects the business entity type (e.g., claims or contracts), and routes them to their respective validation pipelines.
Strict Runtime Validation: Integrated data quality gates (validations.py (http://validations.py)) to validate schema integrity before any database write occurs, preventing corrupt or malformed data from breaking production tables.
Storage Optimization: Implemented a multi-tier storage mechanism where validated datasets are formatted as localized CSV tracking logs and structured as Delta Lake layers (delta_files/, raw_delta) for optimized historical time-travel analysis.
ClickHouse Ingestion: Co-developed high-performance data loading utilities (etl_clickhouse.py (http://clickhouse.py), clickhouse_connector.py (http://connector.py)) to batch-insert processed records straight into ClickHouse, ensuring fast, real-time query responses for analytical dashboards.
Asynchronous Task Architecture: Organized the backend with a detached worker system (celery_project/) to decouple heavy data-crunching and insertion processes from the primary application logic.
1
25
## Project Description
ConvAuto is an automated data pipeline built to handle marketing conversion metrics and analytics in real time. The goal of the project was to bridge backend logic with a cloud data warehouse, making it easy to crunch large volumes of lead data and monitor performance on the fly.
### What I Did:
Dynamic Query Building: Instead of using rigid, static queries, I built a Python-based engine that dynamically generates complex SQL CTEs (WITH ... AS statements). This allows the system to instantly filter and group data by specific dates, buyer IDs, or marketing channels.
Conversion Tracking: Wrote the core business logic to accurately calculate real-time conversion rates (CONV_RATE) and track funded leads. I also made sure it has solid error-handling to prevent issues like ZeroDivisionError when processing empty data sets.
Snowflake Integration: Integrated the pipeline with Snowflake using snowflake-connector-python to ensure fast read/write operations, even when running heavy queries against big datasets.
DevOps & Setup: Fully containerized the app with Docker and Docker Compose for easy local setup, and configured GitHub Actions for automated deployments.
### Tech Stack:
Core: Python, Advanced SQL
Data Warehouse: Snowflake
Infrastructure: Docker, Docker Compose, GitHub Actions
0
19
Production Deployment Automation (VPS + Docker + Nginx)
Implemented a server bootstrap and deployment pipeline for a production backend service.
Includes automated setup of Docker & Docker Compose, Nginx reverse proxy configuration, firewall hardening (UFW), SSL preparation (Certbot), and scheduled database backups via cron.
Designed for reproducible infrastructure setup and zero-downtime service deployment on Linux VPS.
0
52