EvalsLLM evaluationRAG evaluation

RAG Evaluation: Metrics and Test Design

RAG evaluation checks retrieval and generation separately. Learn the core metrics, how to design test sets, and how to tell which part is failing.
Nov 14, 20268 min read
Book a Demo
RAG Evaluation: Metrics and Test Design

TABLE OF CONTENT

SHARE THIS ARTICLE

TL;DR

RAG evaluation measures a retrieval-augmented generation system in two halves: did the retriever find the right information, and did the generator use it faithfully to answer the question? Score retrieval with context precision and recall, generation with faithfulness and answer relevancy, and the whole system with answer correctness against a reference. Build a test set that includes unanswerable questions, conflicting documents and multi-part questions, not just easy lookups. When a score drops, the two halves tell you which part to fix.

Why RAG needs its own evaluation

A RAG system looks up documents and then writes an answer from them. That gives it two separate places to fail. The retriever can bring back the wrong documents, or miss the one that matters. The generator can ignore good documents, misread them, or add claims they don’t support. A single “is the answer good?” score can’t tell these apart, so it can’t tell you what to fix.

There’s a subtler trap too. Faithfulness and correctness are different things: an answer can be perfectly faithful to a retrieved document that is itself wrong or out of date (RAG survey, arXiv). Good RAG evaluation checks both.

This post covers the RAG-specific layer. For evaluation in general, start with our guide to AI agent evaluation.

Where to measure

where to measure a RAG system

Think of it as three measurement points. Retrieval metrics look at the chunks the retriever returned. Generation metrics look at the answer in light of those chunks. End-to-end metrics compare the final answer with a known correct one. Most guides call the core trio of context relevance, faithfulness and answer relevance the RAG triad (CustomGPT).

The core metrics

MetricHalfWhat it asksNeeds a reference answer?
Context precisionRetrievalAre the relevant chunks ranked near the top?Sometimes
Context recallRetrievalDid retrieval find everything the answer needs?Yes
Context relevanceRetrievalHow much of what was retrieved is on topic?No
FaithfulnessGenerationIs every claim in the answer supported by the retrieved chunks?No
Answer relevancyGenerationDoes the answer address the question asked?No
Answer correctnessEnd to endDoes the answer match the known correct answer?Yes
Citation accuracyGenerationDo cited sources actually contain the cited claim?No
Noise sensitivityBothDoes the answer stay right when irrelevant chunks are mixed in?Yes

Context precision rewards the right ordering, not just the right documents, which is why it tends to track end-to-end quality more closely than plain precision and recall (PremAI). Faithfulness is usually computed by splitting the answer into claims and checking what share the retrieved context supports.

The metrics that don’t need a reference answer are the ones you can also run on live traffic. Context recall and answer correctness need known answers, so they belong to your offline test set.

Is it the search or the answer that's failing?

Start Free

Classic retrieval metrics

If you have labeled relevant documents for each question, the classic search metrics still help: precision@k and recall@k (share of the top k results that are relevant, and share of all relevant documents found), mean reciprocal rank (how high the first relevant result appears) and NDCG (which rewards putting the best results first). They tell you whether results are on topic, but not whether the model can actually use them to answer (Manning). Our guide to contextual relevance and other RAG metrics covers retrieval metrics in more depth.

Precision or recall: which matters more?

It depends on the cost of each mistake. In high-risk domains such as healthcare, law and finance, precision usually comes first, because an irrelevant or wrong chunk can mislead the answer (Agenta). For research assistants and broad questions, recall matters more, because a missing document means a missing part of the answer.

Designing a RAG test set

Most RAG test sets are full of easy lookups: one question, one obvious document, one answer. Real users are messier. Cover these case types:

Case typeExampleWhat it tests
Simple lookup“What’s the refund window?”The basics work
Multi-part“What’s the refund window, and does it apply to sale items?”Retrieval of several chunks; complete answers
Multi-documentA question whose answer is split across two policiesRecall; combining sources
UnanswerableA question the documents don’t coverWhether the system says “I don’t know” instead of guessing
Conflicting sourcesAn old and a new version of the same policyChoosing the current source
Vague or misspelled“refnd for gift??”Robust retrieval
Out of scopeA question about a competitor’s productStaying on topic
AdversarialA document with hidden instructions, or a request to reveal internal notesResistance to injection through retrieved content

For each case, record the question, the chunks that should be retrieved (where you can), the expected answer or answer rules, and which metrics apply. Unanswerable cases deserve special attention: a RAG system that confidently answers questions its documents can’t support is a hallucination machine with good manners.

The adversarial row matters more than most teams expect. Documents in the knowledge base, emails or web pages can carry hidden instructions that the generator treats as commands. How to prevent prompt injection in AI agents shows how these attacks work, and how to build an AI red teaming dataset for agents covers how to turn them into test cases.

How many cases?

Start with 30 to 50 covering every case type above, then grow from real failures. For customer-facing systems, pair automated metrics with human review of a representative set; one practical guideline is 100 to 200 queries (CustomGPT). Our guide to golden datasets for AI evaluation explains how to version and grow the set.

Diagnosing failures

The value of separating retrieval and generation shows when something breaks. Read the two together:

diagnosing RAG failures

A quick rule: if faithfulness is high but the answer is still wrong or irrelevant, the retriever brought back the wrong material (PremAI). If the right chunks were retrieved but the answer adds or changes facts, the generator is the problem. Fix retrieval first when both are broken, because a generator can’t be judged fairly on the wrong context.

Evaluating RAG in production

Once the system is live, run the metrics that don’t need a reference answer on a sample of real traffic: context relevance, faithfulness and answer relevancy. Add three cheap production signals: the share of questions where retrieval returns nothing useful, the share of “I don’t know” answers, and user feedback such as thumbs-down or rephrased questions. When the knowledge base changes, rerun your offline set, because new or edited documents can change answers without any code change. Online vs offline evaluation for agents explains how the two connect.

When RAG is part of an agent

In an agent, retrieval is usually one tool among several. The same metrics apply to each retrieval step, but you also need to check whether the agent decided to search at all, searched for the right thing, and acted correctly on what it found. A faithful summary of the right policy is no use if the agent then issues a refund the policy forbids. AI agent evaluation metrics that matter covers the agent-level checks.

Score every RAG answer before release

Start Now

Common mistakes

  • Scoring only the final answer. You’ll know quality dropped but not why.
  • No unanswerable questions. You won’t find out the system makes things up until a user does.
  • Treating faithfulness as correctness. Faithful to an outdated document is still wrong.
  • Never re-testing after content changes. Knowledge-base updates change behavior as much as prompt changes.
  • Trusting judge scores without a human check. Calibrate the judge on a sample first; see LLM-as-a-judge.

The bottom line

A RAG system can fail in the search or in the writing, and one overall score won’t tell you which. Measure both halves, design a test set with the questions that actually break RAG systems, including unanswerable, conflicting and adversarial ones, and use the two halves together to decide what to fix.

BotGauge LLM evaluations scores RAG answers with LLM evaluators and custom criteria, and automated red teaming for AI agents tests what happens when the documents themselves are hostile.

Think your agent is safe? Prove it. BotGauge tests your RAG agent against real attacks. Try Botgauge for Free or book a 30-minute demo.