The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
The timezone bug that broke my data quality pipeline — and how I fixed it
I was building a data freshness check for a Kimball star schema warehouse — one of the validation rules in my Data Quality Checker project. The logic was simple: if the most recent record is older than 24 hours, flag it as stale.
It worked perfectly on my local machine.
Then I tested it on a UTC sandbox and it started failing — flagging fresh data as stale. The timestamps were correct. The records were there. But the freshness check kept returning wrong results.
The bug: my local machine runs IST (UTC+5:30). The freshness check was comparing record timestamps against datetime.now() — which on my machine returned IST local time. On a UTC server, the same function returned UTC. A 5.5-hour gap was making fresh data look like it was older than it actually was.
The fix was straightforward once I found it: replace datetime.now() with datetime.now(timezone.utc) everywhere timestamps were compared. Then I verified the fix ran correctly on both a UTC sandbox and my IST machine before shipping.
Small bug. Easy fix. But it's the kind of thing that silently corrupts your data quality reports in production – and you'd never catch it unless you tested across timezones.
If you're building data pipelines that will run on cloud infrastructure, always assume UTC at the system level and be explicit about timezone conversion at every point where timestamps are compared.
The full project — including the pytest suite, Terraform-provisioned AWS S3, and GitHub Actions CI/CD — is live at https://github.com/Krupa03/data-quality-checker
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Related posts
LCP at 1.9s and CLS at 0 on a mobile Webflow build proves the point better than any argument. What gave you the biggest speed win, image sizes or trimming scripts?
A pipeline can return clean, well-formed, confident output and still be wrong, because it only saw part of the input.
I wrote up a postmortem on a contract-extraction tool built around that failure. A language model pulled renewal dates and notice periods from vendor contracts. A temporary character limit, added to keep test runs cheap, quietly cut off long documents before they reached the model. An amendment near the end of one contract changed the notice period, and the tool never saw it. No errors, no warnings. The model did what it was asked with the text it was given.
The part worth taking from it: most evaluation asks “is the extracted value correct?” Far fewer setups ask “did the system see the whole document?” Those are different questions, and a healthy-looking dashboard only answers the first.
Cheap guards that help:
- Fail loudly when input exceeds the limit instead of slicing it
- Log how much of each document was sent, next to its full length
- Make the model cite the passage each field came from
- Put the hardest cases, long documents with the key clause at the end, in the test set
Not magic: AI + human verification
Every extracted value keeps the text it actually says on the invoice, plus a confidence flag. You check only the doubtful lines - 2 minutes - instead of re-typing everything.
This is the part most AI extraction tools skip, and it is the reason the output can be trusted: a missing date or a line cut off in a photo is flagged for a human instead of being quietly filled in with a plausible-looking value.
What you get: one .exe that runs on your Windows (no subscription, no account), your data never stored on my side, and exports to Excel (.xlsx), CSV, Google Sheets or your ERP.
