Experience in building scalable and efficient pipeline