LLM-as-a-judge: how reliable it is, and where agents need something else

The short answer

“LLM-as-a-judge” means asking a strong model to grade another model’s output. The 2023 study that put numbers on it, by Lianmin Zheng and colleagues, found that strong judges such as GPT-4 agreed with human preferences more than 80% of the time, about as often as people agree with each other.¹ It also measured biases: when the two answers being compared were swapped, GPT-4 gave the same verdict only 65% of the time, and weaker judges much less.¹

For grading open-ended text, a carefully prompted model judge is a reasonable tool. For grading what an agent did, prefer a deterministic check of the system’s state. Agent benchmarks such as AgentDojo and τ-bench do exactly that, and AgentDojo’s authors give a reason: a model judge could itself be hijacked by an injection.²³

What the study did

The paper introduced two resources for evaluating chat assistants on open-ended questions, which fixed-answer benchmarks measure poorly: MT-Bench, a multi-turn question set, and Chatbot Arena, a crowdsourced platform where people compare two models’ answers.¹ The authors then asked whether a strong model could stand in for those human judgements, and tested how it fails.

The biases, in numbers

Position bias. Show a judge answers A and B, then B and A. A fair judge picks the same winner both times. With the default prompt:¹

Verbosity bias. The authors padded answers with repetitive lists that add length but no content. Claude-v1 and GPT-3.5 preferred the padded answer in 91.3% of 23 cases; GPT-4 in 8.7%.¹ Longer looked better to weaker judges.

Self-enhancement bias. Judges favoured their own model’s answers in the data, but the authors are careful: their data can’t determine whether self-enhancement bias is real.¹

Limited reasoning. On 20 math questions where the judge had to tell a right answer from a wrong one, GPT-4 with the default prompt got 14 wrong.¹ A judge that can’t solve the problem can’t grade the solution. That matters for agents, whose work often involves arithmetic, dates, ids and amounts: exactly the details a judge skimming a transcript is least likely to recompute. A refund for the wrong amount reads as fluently as one for the right amount, and only a check against the source data tells them apart.

Fixes that worked

The paper tested several mitigations:¹

The last row is the most important one. Once the judge has a reference answer, it’s no longer judging from taste; it’s comparing against ground truth. That’s a step toward the state checks agent benchmarks use, and it suggests a simple rule of thumb: the more of the answer you can write down in advance, the less you need a model’s opinion at all.

Why agent actions need a different grader

An agent’s output isn’t only text. It’s a set of actions with effects: rows changed, messages sent, files written. A model judge reading a transcript can only see what the agent wrote about those effects. That gives four problems on top of the biases above:

1. The transcript isn’t the effect. An agent can say it updated one record while a trigger changed three hundred.
2. Injection reaches the judge too. If the agent read hostile text, the transcript contains it, and a model judge reads it as well. AgentDojo’s authors chose deterministic state checks partly for this reason.²
3. Verbosity rewards narration. A long, confident account of the work looks better than a short, correct one.
4. Repeatability. A judge that flips on order swaps a third of the time adds noise to every comparison you run.

What works for actions is what agent benchmarks already do: define the expected end state, read the actual state from the system, and compare. τ-bench compares the final database with a goal state.³ AgentDojo uses deterministic functions over the environment.² Neither needs a model’s opinion.

Where a model judge still helps

Model judges are useful alongside state checks, for the parts of an agent’s work that are text:

• Reply quality. Was the message to the customer clear and polite? A judge with examples does this well.
• Policy wording. Did the agent explain a refusal in the required way?
• Triage of failures. Given a failed state check, a model can suggest which step went wrong, for a person to confirm.
• Coverage. Scoring thousands of transcripts for obvious problems, before a person reads a sample.

In each case, use the study’s fixes: swapped positions, examples, reasoning before the verdict, and a reference answer wherever you have one.¹

A checklist for using model judges

1. Use state checks for actions, model judges for text.
2. Swap positions and count only consistent wins.¹
3. Give a reference answer whenever one exists.
4. Ask for reasoning before the verdict.
5. Strip or quarantine untrusted text the agent read before the judge sees the transcript.
6. Spot-check the judge against people on a sample, and re-check after changing the judge’s model or prompt.
7. Never let the judge’s score stand in for a state check on payments, deletions or record changes.

For the state-check side in detail, see how to evaluate agent actions and τ-bench and pass^k. For a benchmark that scores tools against the database’s own change log, see how to benchmark a guardrail. To watch a request compared with what a real Postgres committed, in your browser, open the playground; to grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.