EvalsLLM evaluationLLM-as-a-judge

LLM-as-a-Judge: When It Works and When It Misleads

LLM-as-a-judge can score AI outputs at scale, but judges have known biases. Learn when to trust them, how to calibrate them, and how to judge agents.
Nov 14, 20268 min read
Book a Demo
LLM-as-a-Judge: When It Works and When It Misleads

TABLE OF CONTENT

SHARE THIS ARTICLE

TL;DR

LLM-as-a-judge means using one language model to score another’s output against a written rubric. It works well for questions about meaning, such as relevance, groundedness or policy compliance, and it scales to thousands of outputs a day. In the original study, strong judges agreed with human preferences more than 80% of the time, about as often as humans agree with each other. But judges have known biases: they favor answers in certain positions, longer answers, and sometimes their own model’s answers. Use code checks for anything a rule can decide, calibrate every judge against human labels before trusting it, and re-check it whenever the model, rubric or data changes.

What is LLM-as-a-judge?

An LLM judge reads an output, plus the question and any context or reference answer, and decides whether it meets a standard you’ve written down. Most open-ended AI outputs have no single correct answer, which makes rule-based scoring very hard; a judge model can assess meaning instead (Zheng et al.).

how an LLM judge works

There are two common setups. Single-answer grading scores one output against a rubric, usually pass or fail with a reason. Pairwise comparison shows the judge two outputs and asks which is better, which is useful for comparing prompts or model versions. Both are widely used inside evaluation suites; see our guide to AI agent evaluation for where judges fit alongside code checks and human review.

Scale your evals without trusting them blindly

Start Free

When it works

The case for judges rests on solid evidence. Researchers at UC Berkeley and other universities compared judge verdicts with thousands of expert and crowdsourced human votes, and found that a strong judge matched human preferences at an agreement rate above 80%, the same level at which humans agree with one another (Zheng et al., arXiv). They concluded that LLM-as-a-judge is a scalable and explainable way to approximate human preferences that would otherwise be very expensive to collect (NeurIPS).

Judges are a good fit when:

  • The question is about meaning. Does the reply answer the question? Is every claim supported by the source? Does it follow the policy?
  • There’s no single right answer. Open-ended replies, summaries and conversations can’t be matched against one reference.
  • You need scale. Scoring a steady sample of production traffic, or thousands of test cases on every release, is impractical for people alone.
  • The rubric is clear. Specific, binary criteria give consistent verdicts; vague ones don’t.
  • You have human labels to check against. A judge you’ve measured is a tool. A judge you haven’t is a guess.

When it misleads

The same research documented several ways judges go wrong (Zheng et al., arXiv):

Bias or limitWhat happensHow to reduce it
Position biasIn pairwise comparisons, the judge favors the answer in a particular position, often the first (Emergent Mind)Run each comparison twice with the order swapped; count only consistent verdicts
Verbosity biasThe judge favors longer answers, even when they’re no clearer or more accurate (Emergent Mind)Tell the judge length is not a quality; include short correct answers in your calibration set
Self-enhancement biasThe judge favors answers written by its own modelUse a judge from a different model family than the one being judged
Limited reasoningThe judge misgrades math, logic or facts it can’t check itselfGive it a reference answer, ask it to reason before deciding, or use code checks instead

Later research adds a further warning: judge preferences can favor style over factuality and safety, rewarding answers that read well over answers that are correct or safe (as summarized alongside the original paper by FuguMT). That matters most for exactly the checks teams care about most.

Judges also mislead when:

  • A rule would do. Asking a judge whether a reply is under 100 words, or whether a refund tool was called, adds cost and error to something code can check exactly.
  • The rubric is vague. “Is this a good answer?” gives verdicts that drift from run to run.
  • Nobody checks them. A judge model update, or a rubric edit, can quietly change scores with no change in the system being judged.
  • They’re judging something adversarial. A judge asked whether a reply is safe can itself be influenced by the manipulative content it’s reading. Judges are not a substitute for adversarial testing; automated red teaming for AI agents explains why.

How to calibrate a judge

Calibration means measuring how often your judge agrees with people on your own data, then fixing it until the agreement is good enough to rely on.

calibrating an LLM judge

  1. Label a sample by hand. Pick 50 to 100 real outputs, including clear passes, clear failures and borderline cases. Have someone who knows the domain mark each one against the rubric. For important checks, use two reviewers and settle disagreements.
  2. Run the judge on the same sample. Same rubric, same inputs, no hints about the human labels.
  3. Measure agreement. Count how often the judge and the people agree, and look closely at where they don’t. Separate the two kinds of error: a judge that passes bad outputs is more dangerous than one that fails good ones.
  4. Fix the rubric, not the numbers. Most disagreements trace back to vague wording. Rewrite the pass rule, add an example, or split one check into two, then score the sample again.
  5. Deploy, then keep checking. Once agreement is good enough for the stakes involved, use the judge at scale, and re-check it on a fresh labeled sample regularly, and always after changing the judge model or the rubric.

What counts as good enough depends on the check. For a tone check, occasional disagreement is tolerable. For a policy check on refunds, it isn’t, and a code check on the action is usually better than any judge.

Writing a rubric a judge can follow

Weak rubricBetter rubricWhy
“Is the answer good?”“Does the answer resolve the customer’s stated problem, or clearly explain why it can’t?”Specific and checkable
“Rate helpfulness from 1 to 10”“Pass if every step the customer needs is included; fail if any is missing”Binary verdicts are more consistent than fine scales
“Is it accurate and polite and concise?”Three separate checks, one per qualityOne check per judge call keeps verdicts clean
“Is it grounded?”“Pass only if every factual claim appears in the provided documents”Defines the standard and the evidence

Three more habits help: ask the judge to explain its reasoning before giving a verdict, give it the evidence it needs (the source documents or a reference answer), and include short example verdicts for borderline cases.

Judging agents

Agents add a complication: the thing being judged is a sequence of actions, not just a reply. A polite final message can follow a wrong refund. Three rules make judges useful for agents:

  • Give the judge the trace, not just the reply. It needs to see the tool calls, arguments and results to judge whether the agent did the right thing.
  • Use code for actions, judges for meaning. Whether a refund was issued outside policy is a rule; check it with code on the trace. Whether the agent’s plan made sense for the request is a judgment; use a judge.
  • Judge the whole conversation for multi-turn tasks. Many agent failures, including slow manipulation over several turns, only show up across the full conversation. Multi-turn attacks on AI agents shows what that looks like.

AI agent evaluation metrics that matter covers which agent metrics suit code checks and which suit judges.

Evaluate what your agent did, not just what it said

Start Now

Using judges in production

  • Score a steady sample, plus every high-risk action, rather than everything.
  • Pin the judge model version. If the judge changes, scores change, even when the system being judged doesn’t.
  • Watch the judge, not just the scores. A sudden shift in pass rates with no release on your side may be the judge drifting.
  • Spot-check with people every month, weighted toward borderline and high-stakes cases.
  • Keep adversarial testing separate. Judges score the traffic you have; red teaming finds the attacks you haven’t seen yet. How to build an AI red teaming dataset for agents covers building those cases.

For where judges fit among other scoring methods, see the LLM evaluation metrics cheat sheet and how to build an LLM evaluation framework.

The bottom line

LLM judges are one of the most useful tools in AI evaluation, and one of the easiest to misuse. Use them for meaning, not for anything a rule can decide. Write specific, binary rubrics. Calibrate against people before you trust the scores, watch for position, length and self-preference bias, and re-check whenever anything changes. And never let a judge stand in for adversarial testing.

BotGauge LLM evaluations combines LLM evaluators, code-based checks and custom criteria, and BotGauge LLM red teaming finds the failures judges can’t. For other tools, see the 12 best AI observability and evaluation tools for agents.

Think your agent is safe? Prove it. BotGauge checks what your agent does, not just how its replies read. Try Botgauge for Free or book a 30-minute demo.