The timezone bug that broke my data quality pipeline — and how I fixed it I was building a data f...The timezone bug that broke my data quality pipeline — and how I fixed it I was building a data f...
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Krupa's avatar
Krupa Parmar
• Aug 20
The timezone bug that broke my data quality pipeline — and how I fixed it
I was building a data freshness check for a Kimball star schema warehouse — one of the validation rules in my Data Quality Checker project. The logic was simple: if the most recent record is older than 24 hours, flag it as stale.
It worked perfectly on my local machine.
Then I tested it on a UTC sandbox and it started failing — flagging fresh data as stale. The timestamps were correct. The records were there. But the freshness check kept returning wrong results.
The bug: my local machine runs IST (UTC+5:30). The freshness check was comparing record timestamps against datetime.now() — which on my machine returned IST local time. On a UTC server, the same function returned UTC. A 5.5-hour gap was making fresh data look like it was older than it actually was.
The fix was straightforward once I found it: replace datetime.now() with datetime.now(timezone.utc) everywhere timestamps were compared. Then I verified the fix ran correctly on both a UTC sandbox and my IST machine before shipping.
Small bug. Easy fix. But it's the kind of thing that silently corrupts your data quality reports in production – and you'd never catch it unless you tested across timezones.
If you're building data pipelines that will run on cloud infrastructure, always assume UTC at the system level and be explicit about timezone conversion at every point where timestamps are compared.
The full project — including the pytest suite, Terraform-provisioned AWS S3, and GitHub Actions CI/CD — is live at https://github.com/Krupa03/data-quality-checker
Post image
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started