EvalsLLM evaluationLLM-as-a-judge

LLM Evaluation Metrics: A Cheat Sheet (2026)

A cheat sheet of LLM evaluation metrics: what each measures, when to use it, and which to skip. Covers faithfulness, relevancy, G-Eval, BLEU and more
Nov 14, 20268 min read
Book a Demo
LLM Evaluation Metrics: A Cheat Sheet (2026)

TABLE OF CONTENT

SHARE THIS ARTICLE

TL;DR

LLM evaluation metrics are the scores you use to judge a language model’s outputs: whether they’re correct, grounded, relevant, safe and usable. They differ on two questions: does the metric need a reference answer, and is it scored by code or by another model? For most production apps, the metrics that matter are model-graded ones that need no reference, such as faithfulness, answer relevancy and rubric scores, backed by cheap code checks for format, safety and cost. Classic overlap metrics like BLEU and ROUGE are useful only when there is one right wording.

The two questions that sort every metric

Every LLM metric answers a different question, and most guides list them without showing how they relate. (Agents need more than response metrics; for those, see our guide to AI agent evaluation.) Two properties decide where a metric fits and what it costs to run:

  • Does it need a reference answer? Reference-based metrics compare the output with a known correct response. They’re precise but only work when you can write the right answer in advance. Reference-free metrics judge the output on its own or against the input and context, so they work on live traffic.
  • Is it scored by code or by a model? Code-scored metrics are fast, cheap and exact. Model-graded metrics, usually an LLM judge or an embedding model, can assess meaning, but they cost more and need calibrating.

LLM evaluation metrics by scorer and reference

The top-left corner is where most modern evaluation happens, because open-ended answers rarely have one correct wording. LLM-judge methods such as G-Eval correlate far better with human judgment than BLEU or ROUGE (MetricGate), which is why they’ve largely replaced overlap metrics for generation tasks.

Metrics only catch what your test set covers

Start Free

The cheat sheet

Answer quality

MetricWhat it measuresScored byUse when
CorrectnessIs the answer factually right?Judge against a reference, or exact matchYou have ground truth
Answer relevancyDoes the answer address the question asked?JudgeAny chatbot or assistant
CompletenessDoes it cover everything the question needs?Judge with a rubricMulti-part questions, summaries
Coherence and fluencyIs it logically structured and readable?JudgeMostly to catch degenerate outputs, such as repetition or truncation

Grounding and hallucination

MetricWhat it measuresScored byUse when
FaithfulnessShare of claims in the answer supported by the provided contextJudgeAny RAG or document-grounded app
Hallucination rateShare of answers with unsupported or false claimsJudge, plus human review on a sampleHigh-stakes domains
Citation accuracyDo the cited sources actually contain the claim?Judge or codeApps that show sources

Faithfulness is usually computed as supported claims divided by total claims in the answer (MetricGate). For RAG systems, pair it with retrieval metrics such as contextual precision, recall and relevancy, covered in RAG evaluation: metrics and test design.

Reference and overlap

MetricWhat it measuresScored byUse when
Exact matchOutput equals the expected answerCodeClassification, extraction, short factual answers
F1Token overlap with the expected answerCodeExtractive question answering
BLEUN-gram precision against a referenceCodeTranslation
ROUGEN-gram recall against a referenceCodeSummarization with a reference summary
BERTScoreMeaning similarity using embeddingsEmbedding modelWhen wording may differ but meaning should match
pass@kShare of problems where at least one of k generated programs passes testsCode (unit tests)Code generation

BLEU and ROUGE heavily penalize synonyms and rephrasing (Comet), so don’t use them as standalone metrics for open-ended generation.

Rubric and judge scores

MetricWhat it measuresScored byUse when
G-EvalA rubric score from a judge that reasons through evaluation steps firstJudgeCustom criteria: tone, policy, helpfulness
Pairwise preferenceWhich of two outputs is betterJudge or humanComparing prompts, models or versions

G-Eval was introduced in a 2023 paper and scored far closer to human judgment than reference-based metrics (Liu et al., arXiv). Judges can still be biased toward longer or more polished answers, so calibrate them; our guide to LLM-as-a-judge covers how.

Turn every finding into a scored check

Start Now

Safety

MetricWhat it measuresScored byUse when
Toxicity and biasHarmful, offensive or unfair contentClassifier or judgeAny user-facing app
PII and secret leakagePersonal or confidential data in outputsCode (patterns) plus judgeApps with access to customer data
Injection resistanceShare of prompt-injection attempts the model resistsAdversarial test set, scored by code or judgeApps that read documents, emails or tool output
Refusal calibrationRefuses harmful requests, but not legitimate onesJudge on paired test setsAssistants with safety policies

Safety metrics are only as good as the attacks in your test set. Hand-written injection examples go stale fast; see how to prevent prompt injection in AI agents, and BotGauge LLM red teaming for adaptive attacks that keep the set current.

Format and operations

MetricWhat it measuresScored byUse when
Format validityOutput parses and matches the schemaCodeAny structured output
LatencyTime to first token and to full responseCodeEvery production app
Cost per requestTokens multiplied by model priceCodeEvery production app
ConsistencySame input gives equivalent answers across runsCode or judgeBefore relying on a single test run

Which metrics for which job

If you’re buildingStart withAdd later
A chatbot or assistantAnswer relevancy, correctness on a reference set, toxicityRubric scores for tone and policy
A RAG appFaithfulness, answer relevancy, contextual precision and recallCitation accuracy, hallucination rate
A summarizerFaithfulness, completenessROUGE, if you have reference summaries
An extraction or classification toolExact match, F1, format validityConsistency across runs
A code generatorpass@k, format validitySecurity checks on generated code
An AI agentTask completion, tool-use accuracy, policy adherenceTrajectory scores, consistency, cost per task

Pick three to five metrics per use case, each tied to a failure you actually care about. Our guide on how to build an LLM evaluation framework shows how to turn them into a test suite with pass thresholds.

Common mistakes

  • Using BLEU or ROUGE for open-ended answers. They reward copying the reference’s wording, not being right.
  • Trusting a judge without calibrating it. Check judge scores against a set of human-labeled examples before relying on them.
  • Tracking too many metrics. Twenty dashboards nobody reads catch less than five metrics tied to real failures.
  • Scoring one run. Outputs vary between runs; measure pass rates across several.
  • Leaving safety for later. Injection and data leaks are the failures that make headlines. Test them from day one, ideally with attacks that adapt to your app; automated red teaming for AI agents explains how.

The bottom line

Sort every metric by two questions, whether it needs a reference and whether code or a model scores it, and the long list gets manageable. For most apps, lead with faithfulness, relevancy and rubric scores, back them with cheap code checks for format, safety and cost, and keep overlap metrics for the narrow jobs they were built for.

BotGauge LLM evaluations scores goal completion, accuracy and policy adherence with LLM evaluators, code-based checks and custom criteria. For tool comparisons, see the 12 best AI observability and evaluation tools for agents.

Think your agent is safe? Prove it. BotGauge turns every red-team finding into an evaluation. Start for free or book a 30-minute demo.