Big Data Spark Pipeline
Matthew Robb
Like this project
Posted Aug 1, 2023
Led the architectural design and implementation of a Big Data streaming pipeline.
Led the architectural design and implementation of a real-time big data streaming pipeline capable of ingesting terabytes of data per day and achieving millisecond latency!
The pipeline ingested raw data, monitored quality, applied transformations, obtained predictions from a proprietary model that was deployed as part of the solution. The predictions were formatted and served as per the client’s requirements.