DWH Architecture Diagrams
Real-Time Event Filtering Pipeline (Kafka → Lambda → BigQuery)
Built a pipeline to process millions of Kafka events using AWS Lambda and stream only relevant data into BigQuery.
Instead of moving entire datasets, we filtered events at ingestion:
Lambda triggered by Kafka (event-driven, no polling)
Applied business filters in-flight
Forwarded...