LLM Evaluation Harness — catching regressions before users do

Most LLM systems cannot tell you whether last week's prompt change made them worse. Teams find out from a customer, or they do not find out at all.

Every task is a contract: an instruction, a set of governed lessons, curated examples, and a JSON schema the response must satisfy. Output is validated on the way out, with a single bounded repair attempt before the call fails cleanly.

A failed or malformed call surfaces as a typed error the caller can fall back from. Never a crash, and never a fabricated result papering over a failed call.

Schema validity is structural, though. Catching output that is well-formed but wrong is a different job, and that is what the golden set is for.

A curated golden set of real cases replays before and after every prompt or model change, so behaviour change is measured rather than assumed. Automation may propose a new rule; only an explicit human command activates it. Nothing alters production behaviour without a person approving it.

Hash-chained, metadata-only run traces: IDs, counts and hashes, never prompt or response bodies

Raw prompts and responses stay in a private run store outside the repository

Typed failures the caller can branch on, rather than exceptions to catch

Provider-neutral by construction: task definitions and schemas sit above the provider invocation

I have run this against a CLI-based provider. Wiring an HTTP or SDK provider is a small adapter, and I scope that explicitly rather than assume it. If your stack calls provider SDKs directly, that is a known, bounded piece of work, not a surprise.