Anmol Yadav
Senior Data Engineer | Kafka, Airflow, dbt
New to Contra
Anmol is ready for their next project!
DWH Architecture Diagrams
Real-Time Event Filtering Pipeline (Kafka → Lambda → BigQuery)
Built a pipeline to process millions of Kafka events using AWS Lambda and stream only relevant data into BigQuery.
Instead of moving entire datasets, we filtered events at ingestion:
Lambda triggered by Kafka (event-driven, no polling)
Applied business filters in-flight
Forwarded...