Your AI Agent Doesn't Have a Model Problem Every way an agent fails in production already has a n...Your AI Agent Doesn't Have a Model Problem Every way an agent fails in production already has a n...
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Chetan's avatar
Chetan S
• Aug 15
Your AI Agent Doesn't Have a Model Problem
Every way an agent fails in production already has a name. Most of us were fixing these long before LLMs showed up.
I've lost count of how many versions of this story I've heard in the last year.
The agent works beautifully on someone's laptop. It gets demoed. Everyone in the room is impressed. Budget gets approved, a launch date goes in the calendar, and then it meets actual users — and within about two days something has gone quietly, expensively wrong.
The instinct at that point is to go after the model. Rewrite the prompt. Add guardrails. Try a different provider and see if it behaves better. Sometimes that helps a bit.
But in most of the cases I've looked at, the model was never the problem.
The failures are older than the technology
I spent the first decade or so of my career on telephony systems and enterprise platforms, long before I wrote anything that called an LLM. What strikes me now, looking at how agents fall over in production, is how little of it is new.
Change the vocabulary and almost every agent failure is something the distributed systems world named, argued about, and solved a long time ago.
The agent retries and the customer gets charged twice.
That's duplicate delivery. Anyone who has worked with a message queue has met it. The network doesn't tell you whether your request failed or whether the response just got lost on the way back, so you retry, and now the action has happened twice.
The fix isn't AI-shaped at all. Every operation with a side effect carries an idempotency key. The second attempt is recognised as the same request rather than a new one. That's it. Nothing about a language model changes the shape of that problem.
Step six of a ten-step workflow crashes and all the work vanishes.
There's a pattern people have noticed with long-running agents: they handle short tasks fine, and get much less reliable as tasks stretch out. It's tempting to read that as the model running out of intelligence. Often it's simpler than that — nothing was holding the state.
If each step writes its result somewhere durable before moving on, a crash resumes from step six. If it doesn't, a crash means starting from zero, and every retry burns the same tokens again.
A tool call hangs and the whole run stalls behind it.
No timeout budget. This is the one that takes the rest of the system with it, because the requests pile up behind the stuck one until you run out of workers.
Every outbound call needs a deadline and something that trips when a dependency starts failing. Any team that's run microservices has learned this, usually painfully.
Two agents write to the same record and one update disappears.
Lost update. I wrote about this recently in a completely different context — two people tapping the same booking slot at the same moment. Identical bug, different clothes. The answer is the same too: don't check-then-write in application code, make the database refuse the second one.
Something went wrong and nobody can say which part.
No tracing. Was the latency in retrieval, in the model, in a third-party API three steps down? Without a trace you're guessing, and guessing at 2am is how small incidents become long ones.
Why the framing actually matters
If you file these under "AI problems", you go looking for AI answers. A better model. A smarter prompt. Whichever framework is being recommended this month.
None of that stops duplicate charges.
If you recognise them as distributed systems problems, two useful things happen. You already know the fixes, because they're the same fixes. And you stop treating the situation as some mysterious property of the technology — these are ordinary engineering problems with ordinary engineering answers.
Part of why this keeps catching teams out, I think, is that a lot of the agent tooling was built to make demos work. Demos happen on a good network, with one user, and nothing timing out. Production is the opposite of all three, and infrastructure failure isn't an edge case there — it's Tuesday.
What I put in first
When I add an AI feature to a product now, the reliability layer goes in before I spend any real time on prompts. Roughly:
Idempotency on anything with a side effect. Charging money, sending a message, writing a record.
A queue between intake and processing, with per-conversation ordering. Real users send three messages in a row, and retries arrive out of order.
State checkpointed per step, so a crash resumes rather than restarts.
Timeouts and a fallback response for when the provider is slow or down — because it will be, and a spinner that never resolves is worse than an honest "try again in a moment."
Tracing across every hop, so a bad answer can be walked back to the exact retrieval and prompt that produced it.
A confidence threshold with a human escalation path. Sometimes the correct output is "I don't know," and a system that can't say that will confidently make something up instead.
None of this is clever. It's the same list I'd apply to any service that customers depend on. The only difference with an AI feature is that the failure modes are less visible — a wrong answer looks exactly like a right one until someone checks.
The part I find genuinely interesting
The scarce skill in AI engineering right now isn't prompt engineering. It's operational discipline.
Which leads somewhere slightly counterintuitive: the people best placed to make agents reliable aren't necessarily the ones who've been doing AI the longest. They're the ones who've been woken up at 2am by a pager, and who already have opinions about retries.
If you've shipped an agent into production, I'd like to know what got you first.
My money's on retries. It usually is.
aiengineeringaichatbotdevAWS
Post image
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started