You Changed the Prompt. What Else Did It Break?

Almost every team shipping AI features changes prompts on instinct. Almost none of them could tell you what the last change made worse.

Here's how a prompt usually gets changed in production.

Someone on the support team flags a bad answer. An engineer opens the prompt, reads the conversation, adds a sentence — "if the customer asks about refunds, always mention the 14-day window." They try the question that went wrong. It works now. They try two or three others to be safe. Those look fine too.

It ships.

That process feels responsible. It isn't a test. It's a guess, checked against the handful of cases the person happened to think of, which are always the cases closest to the problem they were fixing.

What actually happens when you change a prompt

A prompt isn't a list of independent rules. Every instruction you add shifts how the model weighs everything else in there.

Tell it to always mention the refund window and it will — including for people who asked about delivery times and now get a refund policy nobody wanted. Tell it to be more concise and some answers that needed three steps quietly lose the second one. Tell it to be warmer and it starts apologising for things that weren't problems.

None of that shows up in the three questions you tried. It shows up a week later, spread thinly across other conversations, where nobody connects it back to your change.

The fix is embarrassingly familiar

I came to AI work from years of building CI/CD pipelines, and the answer here is the one we already use for code: you don't find out what a change broke by looking at it. You find out by running everything you already know should work, before and after, and comparing.

For AI features that's an eval set.

Take a hundred or so real questions from the last month — not invented ones, real ones, including the weird ones. Write down what a good answer looks like for each. Every time the prompt changes, run all of them against the old version and the new one.

Then look at the difference. Not the average score — the difference, case by case.

Because the average lies. A change that fixes twelve answers and breaks five shows up as an improvement. It is one, on balance. But those five customers are now getting worse answers than they got yesterday, and a single number hides them.

Where the eval set comes from

The honest part is that building one is tedious, and that's why most teams don't.

There's a shortcut if you've built the assistant properly, though. If you log every escalation — every question the bot wasn't confident enough to answer — you already have a stream of real, hard, representative questions arriving every week. Some of the best eval cases I've used came straight out of that log.

Your support team also knows exactly which questions matter. Ask them for the twenty they'd be most embarrassed to see answered wrong. Those go in first.

It doesn't need to be large to be useful. A hundred well-chosen cases will catch most regressions. Two hundred is plenty for a support assistant.

What changes once you have one

The obvious thing: you stop shipping regressions you can't see.

The less obvious thing is that it changes how you argue. Without evals, a prompt discussion is opinions — "I think this version sounds better." With them, it's evidence. Someone can propose a change, run it, and show you it fixed nine cases and broke one, and you can decide together whether that trade is worth it.

It also makes switching models a normal decision instead of a frightening one. When a cheaper or faster model comes out — which happens constantly now — you can run your whole set against it in an afternoon and know whether it's good enough for your use case, rather than whether it's good enough on someone else's benchmark.

The part people push back on

"We don't have time to build a test suite for a chatbot."

I understand the instinct. But you're already paying the cost — you're just paying it in customer conversations instead of in a spreadsheet. Every regression that ships is a small, invisible hit to how much people trust the thing, and trust is the only thing that decides whether an AI feature survives its first six months.

A prompt is code. It changes behavior, it has side effects, it can break things that were working. We decided a long time ago that code doesn't ship without tests.

There's no reason prompts should be the exception.

If you're running AI in production — do you have an eval set, or are prompt changes still checked by trying a few questions? No judgement either way. I'm genuinely curious how common this is.