Transform Data in Real-Time with ModelBuilder's Dynamic EngineTransform Data in Real-Time with ModelBuilder's Dynamic Engine
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Subtitle: A dynamic data transformation engine and preview processor designed to parse, map, and process custom metadata schemas at runtime.
Project Description
ModelBuilder is an advanced data modeling backend service built to handle complex, dynamic data transformations. Its core feature is a flexible preview processing engine (preview_processing) that allows users to map, rename, and transform datasets on the fly based on custom payloads and dynamic schemas before committing them to a database.
What I Did:
Dynamic Column Transformation: Developed the runtime mapping logic (processors.py) that extracts and transforms column metadata. The engine dynamically checks strategies like manage, create, or rename to dynamically extract column names from nested dictionary payload structures (col_info['used_cols']).
Metadata-Driven Processing Layer: Built a decoupled processing mechanism (process_columns) that extracts custom values, identifies the required data process type, and instantiates factory classes (processor_class) to process raw data, metadata, and error handling configurations dynamically.
Factory & Modular Architecture: Structured the microservice under a clean, modular layout with factory functions (factory_functions), isolated database models (models/), custom middlewares, and dedicated migration scripts.
Production Deployment Configuration: Set up targeted Docker configurations with both app.Dockerfile (for development) and prod.app.Dockerfile (optimized for production usage), coupled with helper shell scripts (entrypoint.sh, run.sh).
Post image
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started