Hilti Marketing Data System Optimization Project

At Hilti, I rearchitected and developed an efficient data system using Airflow, Databricks, DBT, Athena, and Redshift Spectrum. My role included:

System Re-architecture: Redesigned the entire data processing pipeline for improved efficiency and scalability.

ETL Pipeline Design: Created an ETL pipeline where data from S3 folders is loaded into Databricks notebooks through Airflow, creating Bronze Delta tables.

Transformation Logic Implementation: Utilized Airflow to trigger DBT transformations on Databricks SQL Warehouse, converting Bronze tables to Silver and Gold Delta tables.

Ad-hoc Analytics Setup: Configured Athena to read Bronze, Silver, and Gold Delta tables for ad-hoc analytics.

Data Visualization: Developed processes to load Gold Delta tables into Redshift Spectrum for visualization by end clients.

Performance Optimization: Ensured the new architecture improved data processing efficiency, scalability, and reduced costs.

This new architecture resulted in a streamlined and cost-effective data processing workflow, enhancing overall performance and enabling robust analytics and visualization capabilities.