Let your expert's judgment check thousands of AI outputs, not just a handful.

Why AI quality is hard to see

Every AI system gets things wrong some of the time. The hard part is knowing how often, in which cases, and how badly.

An AI that works in a demo has been tried on friendly examples. Real documents are messy, and real people phrase things in unexpected ways. The gap shows only when real input arrives.

The usual ways of checking quality each leave a gap:

• Spot checks: someone reads a handful of outputs now and then. Rare failures, often the expensive ones, slip through.
• Complaints: a problem surfaces only after a customer has been affected.
• Generic scores: a dashboard shows “relevance 4.2 out of 5”. Nobody can say what separates a 4 from a 3, or what to fix.

This guide uses a narrower and more useful measure: for each output, pass or fail against your own standard, with a reason. A reason such as “wrong delivery date” tells you what to fix; a score of 3.8 does not. Pass or fail is also more reliable than a fine-grained scale, as OpenAI's evaluation guide and Husain both advise. (OpenAI) (Husain, 2024)

The method works for any kind of output. What changes is what “fail” looks like:

What an LLM judge is, and what it is not

An LLM judge is a language model given four things: the output to check, the input that produced it, the facts the output should agree with, and a written standard. It answers pass or fail, and explains why.

The approach is well studied. In a study presented at NeurIPS 2023, strong models acting as judges agreed with human preferences more than 80% of the time, about as often as people agree with each other. Those were general questions, not specialist work like yours. That is why your judge must be tested against your own expert. (Zheng et al., 2023)

The same research documents the weaknesses. A judge can prefer whichever answer it reads first (position bias), favour longer answers (verbosity bias), and rate its own output higher (self-preference). Later work measured the last one: models scored their own output above other output that people rated as equally good. (Zheng et al., 2023) (Panickssery et al., 2024)

So a judge is never taken at its word. It is tested against a person first (Step 7). Preferably it runs on a model from a different provider, of similar strength, so it is less likely to favour its own kind of output.

Not every check needs an AI. Use the simplest one that works:

For actions in other systems, check what actually happened, not what the AI says it did: a rule looks up afterwards whether the right order was cancelled. Run these tests in a test copy of your systems, never on live orders, and start every case from the same clean state.

Automated checks, rules and judges together, are what make it possible to test many cases instead of a few. Anthropic's documentation states the trade-off plainly:

More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals. (Anthropic)

Decide what a mistake costs

There is no perfect AI. The aim is to know how often it fails, how badly, and whether that is acceptable for this use.

Start from the consequences. A poor suggestion in an internal tool is an annoyance. A wrong promise to a customer, a wrong statement about a contract, or a wrong action in another system costs money or trust. The more a mistake costs, the stricter the check, and the more outputs a person should still approve before they take effect.

Write down the three worst things your AI could do, and who would notice. That list tells you where the judge must be strictest and where a person stays in the loop.

Name one expert

The standard has to come from someone who knows your customers, products and rules, not from the software or the people who built it.

Pick the person colleagues go to when they are unsure: a head of customer service, a senior claims handler, a senior account manager. Their judgment becomes the standard the judge is built on and tested against. Hamel Husain, a machine learning engineer who advises companies on AI evaluation, calls this person the principal domain expert, and advises against developers taking the role. Husain calls the method in Steps 2 to 7 critique shadowing. (Husain, 2024)

One person, not a committee. Five reviewers produce five standards, and a judge cannot copy all of them. If several people share the role and disagree, hold a short session first to agree on one standard, and write down the outcome.

Plan their time. In the first weeks they grade the first cases, then the larger set needed to test the judge (Step 7), and go through the cases where the judge disagrees with them. After that they grade a regular sample, plus a new round whenever the AI changes.

Collect real cases

Your expert judges the AI one output at a time, so the cases you pick decide what you learn.

Start with real inputs and outputs, not examples written for a demo. Real cases show problems that test examples miss. The best sources are the ones that already point at problems:

• complaints and escalations
• outputs a colleague corrected before they went out
• conversations where the customer asked the same question twice
• a sample of ordinary daily traffic, so you also see what works

Then add difficult cases on purpose: a skewed, partly unreadable scan; a contract with an unusual clause; a customer with three open orders asking “where is my package?”; a request the AI should refuse or hand over to a person.

How many? A practical start is around 30 cases. Keep adding until new cases stop showing new kinds of failure. Where real data is thin, written test inputs can fill gaps, but real cases come first. Testing the judge later (Step 7) takes more graded cases: enough that the cases kept aside hold a few dozen failures and as many passes. (Husain, 2024)

If inputs and outputs are not saved today, start now, so you have real cases to choose from. Decide at the same time how long you keep them and who may see them. (OpenAI)

Grade each case: pass or fail, and why

Your expert reads each case and gives one verdict, pass or fail, with a short written reason.

The reason matters more than the verdict. It should say what is wrong and what the right output would have been, specifically enough that someone else could apply the same judgment.

Being too terse is a common mistake. (Husain, 2024)

After the first round, group the failures by cause: wrong facts, missing information, a promise the company can't keep, the wrong tone, the wrong action. The groups and the expert's reasons become a short written standard, in your own words.

Fix the obvious before automating

The first round of grading usually shows a few failures that occur again and again.

Fix those first: a missing instruction, an outdated policy document the AI reads from, a field the extraction never fills. Then grade a fresh set of cases. Building an automated judge for problems you can already see is wasted effort.

The judge is for what remains: failures that are rare, subtle, or that come back after a change.

Build the judge

The judge is a set of written instructions for a language model. Its content comes from Steps 3 to 5.

The instructions contain:

• the written standard: what counts as a pass and what counts as a fail, in plain words
• the facts the output must agree with: the delivery policy, the product sheet, the original document behind a summary, or the expert's correct answer where there is one
• a selection of graded cases with the expert's reasons, as examples
• the case to judge: the input and the AI's output
• a request to give a short reason with every verdict

A model from another provider is less likely to favour the kind of output it produces itself, so use one where you can. If that isn't possible, use the strongest model available, and let Step 7 show whether it works.

In practice a developer sets this up. The cases sit in a spreadsheet or an evaluation tool, and a small script sends each case to the judge model and records its verdict and reason next to the expert's. A first version is usually a few days of work.

Start with one judge for the overall verdict. Add separate, narrower judges only where one kind of failure needs its own attention, for example “promises only what the policy allows”.

Split the graded cases into three piles: a few to show the judge as examples, a working set to improve its instructions against, and a final test set that nobody uses until the instructions are finished. The judge must not see the final test set; it is for Step 7.

Test the judge against your expert

A judge is useful only if it agrees with your expert on cases it has not seen.

Run it on the working set and compare its verdicts with the expert's. Look at two numbers, not one:

• Of the outputs the expert failed, how many did the judge also fail? This shows whether problems are caught.
• Of the outputs the expert passed, how many did the judge also pass? This shows whether good work is wrongly flagged.

Decide how high is high enough before you run the test, and write it down: how many of the expert's failures the judge must catch, and how many good outputs it may wrongly flag. Base it on Step 1. For a low-risk internal tool, catching most failures may be enough. Where a mistake costs money or trust, the judge should catch nearly all of them, and a person should still approve the highest-impact outputs.

Make sure the final test set includes enough failures and enough passes, a few dozen of each if you can. With only five failures, one missed failure changes the result by 20 points. (Husain, 2024)

If the numbers are too low, read the cases where judge and expert disagree. Usually the standard is unclear or an example is missing. Adjust the instructions and run again on the working set. When the instructions are finished, run the final test set once. That result is the one you report, because the judge has never seen those cases.

Keep it running

AI quality changes when the provider updates its model, when your instructions or data change, and when customers start asking different things.

The same judge serves several purposes:

A judge can also run live: it checks every output before it goes out and holds back the ones that fail for a person to review. That adds time and cost to every answer, so keep it for outputs where one bad answer does real damage, and use rules there first.

Keep the judge itself honest. Fix it to one model version. Providers publish dated versions; use one of those, not a name that always points to the newest model. When you change that version or its instructions, test it against the expert again before you trust new reports. Have the expert grade a small sample regularly and compare again, and add new kinds of failure to the case set as they appear.

Common mistakes

• Scoring on a 1–5 scale instead of pass or fail with a reason.
• Letting the developers who built the AI decide what good means.
• Grading only demo cases, not real and difficult ones.
• Testing the judge on the same cases used to write it.
• Reporting one agreement percentage when failures are rare.
• Using an AI judge where a simple rule would do.
• Using the same model to produce the output and to judge it, without testing whether it favours its own work.
• Building the judge once and never testing it again after a change.
• Letting the judge's own model change without testing it again.

Proof for customers and regulators

A tested judge also leaves a record: what was checked, how, and with what result.

For AI systems classed as high-risk under the EU AI Act, providers must test the system against metrics and thresholds set in advance. They must reach an appropriate level of accuracy and state the accuracy metrics in the instructions for use. And they must monitor the system's performance throughout its lifetime. (AI Act, Art. 9) (AI Act, Art. 15) (AI Act, Art. 72)

If you use a high-risk system bought from a supplier, you are a deployer. You must monitor it according to its instructions for use, and keep the logs it generates for at least six months, as far as those logs are under your control. (AI Act, Art. 26)

A judge, with its own test against your expert on file, plus its regular reports, is a practical way to produce that evidence, and customers and auditors can read it too. It is a technical measure, not legal advice. Whether your system is high-risk, and from when the obligations apply, is covered in the AI Act guide.

What this would look like for you