Evaluating Generative Systems
Judging output that has no single right answer, without shipping on vibes.
No Answer Key
What replaces it is a layered approach: a fixed evaluation set you can rerun, automated judgement for scale, human review for the questions that matter, and online metrics for what users actually do. Skipping straight to "it looks good in the playground" is how teams ship a regression they cannot see.
Offline Evaluation Sets
Build a fixed set of inputs with expectations attached, and run it on every prompt change, model change and retrieval change. It is the regression suite for a system that has no unit tests.
| Include | Because |
|---|---|
| Typical requests | The common path must not regress |
| Known failures | Every bug found becomes a permanent case |
| Adversarial prompts | Injection attempts and jailbreaks |
| Out-of-scope questions | Refusing correctly is a requirement too |
| Edge cases | Empty input, enormous input, another language |
| Check type | Example | |
|---|---|---|
| Deterministic assertion | Valid JSON, required fields, no PII, under a length | |
| Reference comparison | Contains the expected figure or citation | |
| Graded judgement | Faithful to the source, helpful, appropriate tone |
Get as much as possible into the first category. A deterministic check — schema valid, forbidden phrase absent, citation present — is fast, free and never disagrees with itself.
LLM-as-Judge
Using a strong model to grade output scales far past human review, and it works — with known biases you have to control for.
| Bias | Mitigation |
|---|---|
| Position bias — prefers the first option | Randomise order, run both ways |
| Verbosity bias — prefers longer answers | Score criteria separately from length |
| Self-preference — favours its own family | Judge with a different model than you generate with |
| Vague criteria produce vague scores | Ask for specific yes/no judgements, not "rate 1–10" |
Pairwise comparison — "which of these two is better, and why" — is markedly more reliable than absolute scoring. And the judge itself needs validating: grade a sample by hand, measure agreement, and treat a judge that disagrees with humans as a broken instrument.
Human Review & Online Metrics
Offline sets catch regressions on cases you thought of. Production tells you about the ones you did not.
| Signal | Reveals |
|---|---|
| Thumbs up/down | Cheap, sparse, biased toward the annoyed |
| Edit distance on accepted output | How much users had to fix — a strong quality proxy |
| Task completion | Whether the feature actually helped |
| Retry and rephrase rate | The answer failed without anyone reporting it |
| Escalation to a human | The clearest failure signal in support contexts |
| Cost and latency per request | Quality is not free; both belong on the dashboard |
Treat prompts, models and retrieval settings as versioned artefacts: pin the model version, keep prompts in git, log which combination produced each response, and run the evaluation set before shipping a change to any of them.
Interview Questions
Why is evaluating generative output hard?
There is no single correct answer to compare against, output is non-deterministic, and provider model updates change behaviour without any change on your side.
What goes into an offline evaluation set?
Typical requests, every known failure, adversarial prompts, out-of-scope questions and edge cases — rerun on every prompt, model or retrieval change.
What are the biases of LLM-as-judge?
Position bias toward the first option, verbosity bias toward longer answers, and self-preference for its own model family. Randomise order, score criteria separately, and judge with a different model.
Why prefer pairwise comparison to absolute scoring?
Models are much more consistent at saying which of two outputs is better than at assigning a stable number, where scores drift between runs.
Which online signal is most useful?
Edit distance on accepted output — how much the user had to change it — plus retry and escalation rates, which capture failures nobody bothers to report.
Why version prompts and pin model versions?
Both change behaviour. Without pinning and logging which combination produced a response, a quality change cannot be attributed to anything.