I Spent Years Shaving Milliseconds. Then We Added an LLM.

Most of the wait isn't the model. It's everything you built around it.

For a long stretch of my career, latency was the job.

On the VoIP platform, a few hundred milliseconds of delay in the signalling path was the difference between a call that felt natural and one where people talked over each other. We measured everything. We argued about connection pools. A regression of 80ms was a conversation.

Then we started putting language models into products, and suddenly a nine-second response was considered normal. Nobody flinched. The demo worked, the answers were good, and the wait was just accepted as the cost of doing AI.

It isn't. And the thing I found most interesting when I actually measured one of these end to end is that the model was not the main problem.

Where the time actually goes

If you profile a typical AI assistant request, you get something like this.

A small amount at the front for auth and routing. Then retrieval — and this is usually where the first surprise is, because most implementations run their lookups one after another. Search the vector index, then fetch the conversation history, then pull the customer record. Each is fast. In sequence they're not.

Then prompt assembly, which sounds instant and often isn't, because someone is rebuilding the same system prompt and the same set of examples on every single request.

Then the model call itself.

Then post-processing — parsing, validating, formatting, maybe a second model call to check the first one — which happens after the model has already finished, while the customer continues to wait.

In the one I measured, the model was under half the total. Everything else was mine.

The fixes are unglamorous

Run retrieval in parallel. Your vector search, your conversation history and your user lookup don't depend on each other. Fire them together. This is first-year concurrency and it routinely saves seconds.

Cache the parts of the prompt that never change. Your system instructions and your few-shot examples are the same on every request. Assemble them once. Several providers will also cache a common prefix on their side and charge you less for it, which is a rare case of faster and cheaper being the same change.

Overlap post-processing with generation. If you're streaming, you can start validating and formatting the beginning of the answer while the end is still being produced. Most implementations wait for the whole thing first, out of habit rather than necessity.

Don't use your biggest model for everything. Intent classification doesn't need the same model that writes the final answer. A smaller model on the routing step is faster and cheaper, and the quality difference on a classification task is usually nothing.

The one that matters more than all of them

Stream the response.

Nine seconds where nothing happens is unbearable. Nine seconds where words start appearing after 800 milliseconds is completely fine — and people will read along as it generates, which means the perceived wait is closer to one second than nine.

The total time is identical. The experience is not remotely the same.

This is the single highest-leverage change available in most AI features, and it's often skipped because the first version was built as a request/response call and nobody went back to it.

Why this is worth caring about

There's a comfortable assumption that this problem solves itself — models keep getting faster, so wait it out.

Partly true. But inference speed improving doesn't fix serial retrieval, and it doesn't fix rebuilding your prompt from scratch every time, and it doesn't fix blocking post-processing. Those are architecture problems and they'll still be there on a faster model. You'll just have a shorter blue bar in the middle of the same chart.

And the cost of slowness is real. People abandon. On a support assistant, a slow answer pushes customers to ask for a human, which is precisely the outcome the system was built to avoid — and it looks in your metrics like the bot failed to answer, when actually it just answered too late to be wanted.

The bit I keep coming back to

The habits that made me useful on telephony systems turned out to transfer almost completely. Measure the whole path, not the part everyone's looking at. Parallelise what doesn't depend on anything. Cache what doesn't change. Show the user something as soon as you possibly can.

None of that is AI knowledge. It's just performance work, applied to a new place where — for reasons I don't fully understand — we all agreed to stop doing it.

Have you actually profiled one of your AI features end to end? I'd be curious what fraction turned out to be the model versus everything around it. Mine was under half, and I was genuinely surprised.