The cheapest model on the price list isn't the cheapest to run
We ran the same 25 synthetic invoices through every candidate model:
The cheapest listed model answered "rate-limited upstream" on 15 of 25, even when run alone. A model that doesn't answer costs a person's time.
The...
Things that broke in production (and the checks they became)
Backups that backed up nothing. The script dumped an empty Postgres database while n8n was actually running on SQLite, and the restore test didn't catch it. Now there are online SQLite snapshots, counts read before...
Re-measure what you publish
Two numbers I had to correct on my own site:
The AI receptionist's lead decision was published as 2.8 s, from a one-day benchmark. On 281 real leads the median is 3.0 s and the p95 is 12 s (one model misses its deadline and the next one answers).
The...
Sorting a review queue by "worst reading first" hid the fraud
Docket's review queue used to sort by the worst extraction first. We replayed it on 122 waiting documents where the right answer was known. The money-at-risk ones (duplicates, bank-detail changes, PO mismatches) sat...
One real document beat 100 synthetic ones
Dossier caught 100% of our synthetic forged applicant packs with zero false alarms. Then the first genuine UK bank statement arrived by email, and it was flagged as "balances don't add up".
Every row did add up. The problems were all...