Orchestration
What is an AI judge?
An AI judge is a model that scores other models’ answers. How judging is used, the biases it carries, and why a judge is not a substitute for a real check.
All explainers · Last reviewed:
What a judge does
A judge receives the original question and two or more answers, and returns a ranking or a score, usually with a reason. It is the selection step in best-of-N, and the arbitration step when models disagree.
The appeal is obvious: it scales in a way human review does not, and it is far cheaper than the person it stands in for.
The biases, which are consistent
Judges favour longer answers over shorter ones, even when the shorter answer is complete. They favour confident phrasing over hedged phrasing, regardless of whether the confidence is warranted. They are sensitive to the order candidates are presented in. And a model asked to judge its own output tends to prefer it.
None of these is subtle, and all of them are manageable — shuffle the order, use a different model as judge than as author, and ask for specific criteria rather than a general preference. But they have to be managed deliberately, because the default setup exhibits all four.
A judge is not a verifier
A judge compares answers to each other. It does not compare them to reality. Given three wrong answers it will confidently rank them, and the winner will still be wrong.
Where an external check exists — tests, a schema, a search — that check beats a judge, because it is independent of the thing being judged. A judge is what you use when no such check is available.
Questions people ask
- Should the judge be the strongest model?
- Usually a strong one, and preferably not the same model that wrote the candidates. Self-preference is real and the cheapest fix is to use a different model.
- Can a judge score a single answer?
- It can, but comparative judgement is more reliable than absolute scoring. Models are better at "which of these is better" than at "is this a 7 or an 8".
- How do I know the judge is right?
- Spot-check it against your own judgement on a sample. If you never check, you have moved the trust rather than earned it.
Try it rather than read about it
ClawAI runs judging as its own surface over a compare run, so a scored answer records both the models that wrote the candidates and the model that judged them.