What are AI evals? Testing answers that can change each time

A benchmark score is not your product's test

A broad benchmark can compare models on a shared collection of tasks. An application eval asks a narrower question: can this assistant answer our policy questions correctly using the information and tools we give it? A strong general score does not directly answer that.

Define the task before choosing a score. For a policy assistant, useful criteria might be: preserve the policy's conditions, answer the customer's question, and avoid inventing an entitlement. These criteria are more informative than ‘sounds helpful’.

[1]

Worked example: a fictional returns assistant

Suppose a made-up shop accepts unused items within 14 days. Damaged arrivals go to a support agent for review. The assistant can explain the policy but cannot approve refunds. We test three cases against that policy, keeping the policy available to the system.

For an unused item bought ten days ago, both ‘It is within the return window if unused’ and ‘Unused items qualify for 14 days, so yours is within that period’ can be acceptable. The wording differs; the decision and condition agree. ‘You definitely have an approved refund’ fails because the assistant has no approval authority.

How do you grade different but valid answers?

Exact matching works when the required output really is exact, such as a permitted category label. It is brittle for natural explanations: it can reject an accurate paraphrase. For the returns example, grade the decision, conditions and unsupported promises separately.

Humans can use a written rubric with examples of passes and failures. An AI judge can help apply that rubric at scale, but its verdict is another model output that can be wrong. Compare its decisions with human-reviewed examples, inspect disagreements and revise the rubric when people cannot apply it consistently.

For an assistant that takes actions, check the resulting state too. Saying ‘I cancelled it’ is not evidence that an order was cancelled. A good-sounding final message and a successful action are separate things to verify.

[2]

Misconception: 90% means it is ready

Imagine a ten-case trial passes nine cases and invents refund approval on the tenth. Calling it ‘90% accurate’ hides the nature of that failure. Report the unauthorised-promise failure separately instead of allowing many easy successes to obscure it.

Ten handpicked cases also cannot establish a dependable real-world error rate. Coverage, selection and consequences matter. Include normal requests, missing information, exceptions and previously observed failures. Keep some cases aside when improving the system so you are not repeatedly tuning to the same answers.

[1]

Use the result to locate the problem

If the assistant gives a wrong deadline, inspect what it received. Did retrieval supply an outdated document? Did it omit the relevant paragraph? Or did the model receive the correct policy and still contradict it? The final answer can fail for different reasons, and changing the prompt will not necessarily fix the source document.

Run important cases more than once when outputs vary, and record the model, instructions, tools and data used. Compare versions on the same held-out cases. Evals provide evidence about those tested conditions; monitoring real use is still needed to discover cases the test set missed.

[2]