Teams running LLM features in production watch the bill climb and can't say why. Which feature is driving it? Which model? How much goes to retries? Most optimisation advice is a percentage nobody actually measured.

llm-audit is a command-line toolkit I built to answer those questions with evidence.

HOW IT WORKS → Ingest: OpenTelemetry GenAI traces, Langfuse data, or usage exports from OpenAI and Anthropic, loaded into one DuckDB table. → Analyze: 16 deterministic checks, 8 for cost (unused prompt cache, model overkill, duplicate calls, context bloat, retry waste, off-peak eligible work and more) and 8 for reliability (error rates, latency, rate limiting, single-provider risk and more). → Backtest: replays the workload with each fix switched on alone, then compares predicted and measured savings. → Report: one HTML file with ranked findings. Each finding has its evidence query, a fix and a confidence level.

WHAT MAKES IT DIFFERENT • Content-free. It works from prompt hashes and token counts only, so prompt text never has to leave the client. • No unmeasured claims. A saving is either measured by the backtest or labelled as an estimate. A fix that lowers quality is marked rejected, even when it saves money. • Honest gaps. Checks that can't run on the data are listed with the reason, not quietly dropped.

THE PART I'M PROUDEST OF My first paid backtest failed its own checks: 75% recall and 103% savings error. Instead of tuning the numbers, I debugged the harness. The test runs were sharing the provider's prompt cache, so they weren't independent. A fresh marker per run fixed it, and the re-run confirmed the isolation (0.0 cache hits where there had been up to 1.0).

FIELD-TESTED ON MY OWN APP I audited AI Lab, my public Q&A app over the EU AI Act and GDPR. Finding zero: the app kept no per-call record at all. My merged PR added a content-free, per-attempt trace. The audit then showed that the evaluation judges took about two thirds of the spend, so scoring answers cost twice as much as producing them.

BUILT LIKE THE AUDIT IT SELLS 915 tests passing · dependencies pinned with hashes · $0.97 total LLM spend for the whole build, under a $15 cap enforced in code.

The demo workload is synthetic and the field run covers one app, so no percentage here applies to your bill. An audit of your own data is the only way to find out. A good starting point is a Bill Snapshot: send your usage exports and get ranked findings back in 48 hours.

#LLMOps #FinOps #AIEngineering #GenAI #LLM #AICostOptimization #Observability #OpenTelemetry #Python #DuckDB #AIAudit #PrivacyByDesign