Cross-cloud real-time data pipeline for by Paul GCross-cloud real-time data pipeline for by Paul G

Cross-cloud real-time data pipeline for

Paul G

Paul G

Cross-cloud real-time data pipeline for an e-commerce platform processing 2M+ events per day. Event sources feed into SQS queues and AWS EventBridge. Lambda functions handle initial transformation and validation before landing raw data in S3. A cross-cloud sync layer pushes events into GCP Pub/Sub for real-time stream processing. Dataflow streaming jobs handle enrichment, deduplication, and windowed aggregations before passing to Cloud Run microservices for feature engineering. Vertex AI runs real-time prediction models for recommendation scoring and fraud detection. Processed features land in Redis for low-latency serving and BigQuery for analytics. Cloud Storage holds Parquet files for batch retraining. Cloud Build orchestrates the ML pipeline, including model training, evaluation, and deployment.
Like this project

Posted Feb 2, 2026

Cross-cloud real-time data pipeline for an e-commerce platform processing 2M+ events per day. Event sources feed into SQS queues and AWS EventBridge. Lambda ...