Evals - tests for AI features
Unit tests assert exact output; an LLM gives a different answer every time. Evals measure quality over a set of cases instead, so you can compare a new prompt or model against the current one with numbers rather than "it looked fine on two examples".
Building an eval set
- Start with 20-50 real cases: typical requests, edge cases, past failures, adversarial inputs.
- Add every production bug as a new case - the set grows like a regression suite.
- Store expected properties, not exact strings.
How to grade
| Grader | Good for | Example |
|---|---|---|
| Code checks | format and hard rules | valid JSON per schema, no forbidden words, length limit |
| Reference match | extraction, classification | priority === 'high' |
| LLM as judge | open-ended quality | "Does the answer use only the given sources? yes/no + reason" |
| Human review | calibration, high stakes | weekly sample of real conversations |
ts
const cases = [
{ input: 'App crashes on login after update', expect: { priority: 'high' } },
{ input: 'Typo on pricing page', expect: { priority: 'low' } },
];
for (const c of cases) {
const out = Ticket.parse(await classify(c.input)); // schema check
score.add(out.priority === c.expect.priority); // exact check
}
console.log(`accuracy: ${score.rate()}`); // compare with the baselineRun each case several times - one lucky run proves nothing for a non-deterministic system.
Safe changes
- Version prompts in the repo next to the code; review prompt diffs like code diffs.
- Pin model versions instead of floating aliases, so the model does not change under you.
- Run evals in CI on prompt or model changes; block on a drop below the baseline.
- Roll out gradually - A/B or a percentage of traffic, and watch production signals: thumbs down, regenerate rate, latency, cost per request.
A new model being "better on benchmarks" says nothing about your feature. Your own eval set is the only thing that tells you whether a change is an upgrade for your users.
updated
did you know it?