Personal open-source project. A hands-on lab for testing an event-driven data pipeline end to end: a Kafka-fed lake with bronze, silver and gold layers, an automated end-to-end suite, a SQL data-quality suite, and a canary job that injects real defects into the pipeline to prove the checks actually catch them.
The domain is a small multi-tenant shipment-tracking platform. Everything runs locally with one command.
What it demonstrates
E2E testing of an eventually consistent system, with no fixed sleeps
Deterministic, seed-driven test data (same seed, same dataset, everywhere)
Publishing real data through Kafka and HTTP
Realistic messiness: replays, out-of-order events, invalid payloads
Advanced SQL checks: percentiles, reconciliation, gaps, duplicates, tenant isolation, freshness
An independent oracle for aggregates (percentiles recomputed from the seed, not from the database)
Testing the tests: fault injection plus canary assertions
Parallel-safe tests on a shared database
CI with a nightly audit, a job summary and report artifacts
0 likes · 0 views · Nov 10, 2026
Open-source lab: end-to-end and SQL data-quality tests for a Kafka bronze/silver/gold pipeline, plus a canary that injects real defects to prove checks work.