What does your AI feature do when the model is down?

We plan for the database failing. Almost nobody plans for the model.

For a long stretch of my career, the most useful question in a design review was also the most annoying one. What happens when this is down?

Not if. When. The database, the cache, the payment gateway, the telephony provider. Every dependency got that question, and every one needed an answer before it went anywhere near production. Sometimes the answer was "we queue it and retry." Sometimes it was "we show the last known value." Sometimes it was "the feature switches off and the page still loads." The answer mattered less than the fact that someone had decided it.

I've started asking the same question about AI features. The room usually goes quiet.

The model is a dependency you don't run

Here's the pattern I keep seeing. A team builds a support assistant, a summariser, a search box that answers in sentences. The model call sits right in the middle of the request path, and it gets treated like business logic. Part of the product.

It's a network call to someone else's servers. It has rate limits you don't set and a status page you don't write. Every major provider has had bad hours: elevated error rates, slow responses, a model update that quietly changes behaviour. That isn't a criticism of them. It's what third-party dependencies do.

So when it happens, what does your feature do? In most codebases I've looked at, the honest answer is that it hangs for thirty seconds and then shows a generic error. Or it retries in a tight loop and makes the rate limiting worse for everyone.

Nobody chose that. It's just what the code does when nobody chooses.

A second provider is one rung, not the ladder

The first instinct is failover. Wire up a second vendor and switch when the first one fails. I'd still do this for anything that matters. But it covers less than people expect.

Your prompts were tuned against one model. The second one follows instructions differently, formats output differently and refuses different things. If anything downstream parses structured output, the fallback can break your parser in ways the primary never did. I wouldn't trust a fallback model until it had passed the same eval set as the primary.

And failover does nothing when the problem is on your side. An expired key. A monthly budget cap you hit on the 20th. A quota you blew through because of a traffic spike. Your second provider is healthy, and your feature is still down.

Decide the ladder before you need it

What worked for every other dependency works here. Define degraded modes in advance, per feature, in order. For a typical AI feature, I'd start with four.

Rung one, the primary model. The normal path.

Rung two, a fallback model. Tested against the same evals, probably smaller and cheaper. Answers get a little worse. Most users won't notice.

Rung three, no generation, still useful. Return the retrieved documents without the summary. Serve the cached answer to a common question. For a RAG system this rung is nearly free, because retrieval never needed the model in the first place. It's also the rung I most often find missing.

Rung four, an honest message and a manual path. "The assistant is unavailable right now. Here's the help centre, and here's how to reach a person." Fast, clear, and it doesn't pretend.

Then the plumbing that moves you down the ladder. A timeout on the model call set from the latency you actually observe, not whatever the SDK ships with. A circuit breaker, so after a run of failures you stop calling and drop a rung straight away, instead of making every single user wait out the timeout. And a flag someone can flip by hand, because sometimes the provider is technically up and the answers are just bad.

None of this is clever. Circuit breakers have been wrapped around flaky downstream services for a long time. The only new part is admitting the model is one of them.

Test it by pulling the plug

A degraded mode you've never exercised is a guess. Block the provider's hostname in staging and click through the product. Set the timeout to one millisecond and watch what the user sees. It's an afternoon of work. My bet is most teams find at least one screen that spins forever.

"The providers are reliable enough now"

That's the obvious reply, and it's partly fair. The big providers are genuinely good at running these systems.

But from the user's side, an outage isn't only a 500. A latency spike during your busiest hour looks like an outage. So does hitting a rate limit on launch day, or a model update that starts returning answers your parser can't read. And the more of your product sits behind one model call, the more a single bad hour costs you. Better uptime lowers the odds. It doesn't answer the question.

The decision falls between two desks

Here's the part I find most interesting. Rungs one and two are engineering decisions. Rungs three and four are product decisions. What counts as "still useful"? What does the message say? Who does the user get routed to, and is anyone staffed to answer?

Engineering assumes product will decide. Product assumes the model just works. So nobody decides, and the spinner wins by default. If you take one thing from this, make it a thirty-minute meeting with both sides in the room and the ladder on a whiteboard.

Which rung is missing in yours? In the systems I get asked to look at, it's almost always the third one. The useful answer that doesn't need the model at all.