You wouldn’t ship a payment flow without tests. Yet a lot of AI features go live on one argument: it looked good in the demo. Then someone tweaks the prompt, or the provider releases a new model version, and nobody can say whether things got better or worse.
Vibes don’t scale
When you build a feature on top of a language model, the output is no longer deterministic. The same input can give a slightly different answer tomorrow. That doesn’t make testing impossible. It changes what you test: not exact strings, but whether the answer is good enough for the job.
That is what an eval is. A fixed set of realistic inputs, a definition of “good” per input, and a score you can compare between versions.
Start with thirty examples, not three thousand
The biggest mistake I see is waiting for the perfect dataset. You don’t need one. Write down thirty real cases:
- Twenty typical requests your users actually send.
- Five edge cases: vague questions, wrong language, a typo in a product name.
- Five things the feature should refuse or hand over to a human.
For each case, note what a good answer must contain and what it must never contain. That last column is where most of the value hides.
Score what you can, judge what you can’t
Some checks are plain code. Is the output valid JSON? Does it mention the order number? Did it stay under 200 words? Write those as normal assertions.
Other checks are fuzzier, like “is this answer polite and correct?”. For those, a second model with a strict rubric works surprisingly well, as long as you spot-check its verdicts yourself every now and then. An eval you don’t trust is worse than no eval.
Run it on every change
Put the eval in your pipeline next to the unit tests. A prompt change is a code change. A model upgrade is a dependency upgrade. Both deserve a score before they reach production, and both should be able to fail the build.
The first time an eval catches a regression that would have reached customers, the thirty examples have paid for themselves. In my experience that happens sooner than you think.
The real benefit: you can move faster
Evals sound like extra work, but they do the opposite. Without them, every change to an AI feature is a gamble, so teams stop changing things. With them, you can try a cheaper model, a shorter prompt or a new retrieval strategy in an afternoon and know the answer by the end of the day.
That is the same reason we write unit tests. Not to prove the code is perfect, but to make it safe to keep improving it.
