Testing AI agents before you ship them

With normal code, testing is clear. You call a function, you check the result, it passes or fails. Every time.

AI agents are different. Ask the same question twice and you can get two different answers. Change one word in the prompt and a feature that worked yesterday quietly breaks. Swap to a cheaper model and nobody notices the answers got worse until a client complains.

So "it worked when I tried it" is not a test. You need evals.

#What an eval is

An eval is a test set for AI. A list of real inputs, each with a way to judge whether the answer is good. You run the whole list against your agent and get a score. Then you change something, run it again, and compare.

#Step 1: build the test set from real life

The best test cases come from real usage, not from your imagination. I collect:

• Questions real users asked
• Cases where the agent failed before (every bug becomes a test)
• Tricky edge cases: empty input, two clients with similar names, questions with no answer

Fifty good cases beat five hundred made-up ones.

#Step 2: decide how to judge

Some answers can be checked by code. Did it return valid JSON? Did it pick the right client id? Did it refuse to answer from another client's data? These checks are exact and free.

Other answers are about quality. Is this reply draft polite and correct? For those I use a second model as a judge, with a clear rubric:

You are grading an answer about a client account.
Score each from 1 to 5:
- correct: every fact matches the notes
- grounded: nothing is made up
- useful: it answers what was asked
Return JSON: {"correct": n, "grounded": n, "useful": n, "reason": "..."}

A judge model is not perfect, so I check a sample of its grades by hand from time to time.

#Step 3: head-to-head comparisons

When I want to know if a new prompt or model is better, I do not look at two scores in isolation. I run both versions on the same cases and ask the judge which answer is better, case by case. I also shuffle which one is shown first, because judge models tend to prefer whichever answer they read first.

This tells you something a single score cannot: where exactly the new version wins and where it loses.

#Step 4: test conversations, not just questions

Many agent bugs only show up in a conversation. The agent answers the first question well, then forgets the context by the third message, or mixes up two clients mentioned earlier.

So some tests are multi-turn: a short script of several messages, with checks after each step. These catch memory and context bugs that single-question tests miss completely.

#Step 5: reliability runs

Because answers vary, one run is not enough. For important cases I run the same input several times. If an agent gets it right 7 times out of 10, that is not "working". That is a coin flip with extra steps. The score I care about is how often it gets it right, not whether it can.

#The scorecard

Every run produces one simple report:

Cost and speed are on the scorecard on purpose. A version that is 2% better but three times more expensive might not be the right choice.

#When I run evals

• Every time a prompt changes
• Every time I switch model or provider
• Before every release
• Whenever a new bug is found, after adding it as a test case

#What I learned

• Every bug you fix should become a test case, or it will come back.
• Use code to check what code can check. Use a judge model only for quality.
• Compare versions head to head, and shuffle the order.
• Test conversations, not just single questions.
• Measure how often it works, not whether it can work.

An agent without evals is a guess. With evals, changing a prompt or a model becomes a normal engineering decision: run the numbers, compare, ship.