Building efficient data pipelines is not only about handling large volumes of data—it's about doing it with minimal resources and low latency.

In this project, I built a lightweight market data pipeline capable of processing 10+ million market ticks per day while running on only 1GB RAM. Exchange WebSocket Streams ↓ Concurrent Go Parser ↓ Data Validation & Normalization ↓ Apache Parquet (Columnar Compressed Storage) ↓ DuckDB Analytics ↓ FastAPI APIs & Live Dashboard Tech Stack

Python • Go • FastAPI • DuckDB • Apache Parquet • Docker • WebSockets

This project strengthened my expertise in high-performance data engineering, concurrent programming, distributed systems, and real-time analytics.

Currently building scalable Data Engineering, Backend Engineering, and Low-Latency Trading Infrastructure solutions.

#Python #DataEngineering #Go #DuckDB #ApacheParquet #FastAPI #Docker #WebSockets #RealTimeData #Backend #SoftwareEngineering