You can trust an agent with customer money only if the interface shows the cost of its mistakes up front.
A platform where a team ships and supervises an AI agent that issues credits on disputed freight invoices. Concept project with fictional data.
Who it’s for. Six roles around one agent: AI engineer, product manager, domain reviewer, billing specialist, on-call engineer, and compliance.
What I did. Domain research, spec and data model, 9 screens and 11 states on the design system, user flow, and prototype.
The Problem
The agent checks an invoice against the contract and issues credits up to $250 on its own. Its mistakes turn straight into money paid out to customers.
Traces, evals, and rollouts usually live in separate engineering tools. Nobody answers in one place: can this version ship, and what could it cost?
Every screen sits on one loop: a mistake in production becomes a test the next version must pass.
Each Screen Opens With an Answer, Not Data
Release review: v15 can’t go to shadow. Each reason has an owner and a next step.
The headline is the verdict: “v15 can’t go to shadow.” The two reasons have a shape, 2 of 18 critical cases failing and over-credit at +$380, and each sits next to its owner and next step.
Show Where the Next Decision Leads
Rollout monitor: v16 is one rejection away from auto-rollback.
The same screen after one more rejection: 29%, and the system rolled back to v14 on its own.
Canary v16 runs on 10% of traffic. The scale shows three outcomes: 17% now, 12.5% if both pending actions are approved, 29% and a rollback if either is rejected.
Check the Agent in Seconds, No Mental Math
Approvals: invoice, contract, and the agent’s math in one bar. The button repeats the amount.
The amount can only go down. Anything above the agent’s math becomes a manual credit in Billing.
A billing specialist decides on a credit without leaving the screen. The $413 difference shows up as bar length, not arithmetic.
Stop the Agent in One Action
Incident: the agent issued a credit twice, and $585 in damage keeps growing while nobody responds.
The confirmation states the cost of pausing up front: about 11 disputes an hour go to people, and the $585 already issued stays issued.
There is one red button on the screen, next to the damage and the time without a response. Every option, from pausing to taking ownership, shows its consequence, so on-call can choose without opening the docs.
The decision strip started here: what is at stake on the left, actions on the right, a consequence under each. It then moved to release review, rollout, and approvals, as the screens above show.
Where the Cost Comes From
Every agent credit has a trace. One bar shows the math: 3 h 30 min on site, 2 h free, 1.5 h billable, 2.5 h billed. The extra hour is $150 paid out without a human.
Trace: a $150 credit for one overbilled hour of detention, issued automatically.
How a Mistake Becomes a Test
A reviewer checks the agent against the documents and disagrees with the AI judge, so the case goes to judge calibration and the failures dataset.
Datasets: new cases land in a draft, and the next version can’t ship until it passes them.
How Much the Agent Can Do Alone
Autonomy policy: raising the limit from $250 to $400 is shown as a consequence, 117 more credits in 30 days without a human. Only Compliance approves.
Scope
The company, people, and amounts are fictional, and there are no post-launch metrics. Numbers are reconciled across every screen.
The spec, domain research, and data model were built with AI using my method, then checked and refined by me.
I design B2B interfaces where a wrong click costs money.
0 likes · 1 view · Nov 11, 2026
An AI agent issues credits on disputed freight invoices. A concept that shows the cost of its mistakes before anyone clicks.
