Real-Time Containerised Data Pipeline

Tech Stack: Golang, Python, Apache Kafka, Benthos, FastAPI, Docker

● Designed and implemented a real-time data pipeline to efficiently stream and persist "real-time" dummy data into a PostgreSQL

database

● Utilized FastAPI, Benthos, and Apache Kafka to optimize data ingestion and processing speeds, benchmarking performance metrics

for continuous improvement