Big Data Spark Pipeline

Led the architectural design and implementation of a real-time big data streaming pipeline capable of ingesting terabytes of data per day and achieving millisecond latency!

The pipeline ingested raw data, monitored quality, applied transformations, obtained predictions from a proprietary model that was deployed as part of the solution. The predictions were formatted and served as per the client’s requirements.