How to Fully Evaluate a RAG System – Metrics for Retrieval and Generation Stages

Why Stage‑wise Evaluation Is Needed

RAG (Retrieval‑Augmented Generation) is a two‑stage pipeline: first a retrieval module selects relevant document chunks, then a large language model (LLM) generates an answer using those chunks. Because the failure modes of the two stages differ completely, overall performance cannot reveal which stage is problematic. Therefore, evaluation must be split into retrieval and generation phases.

Retrieval‑Stage Metrics

The core questions are: Are the retrieved chunks truly relevant? Are any relevant chunks missed? Is the ordering reasonable? Precision measures the proportion of retrieved chunks that are relevant, while Recall measures the proportion of all relevant chunks that were retrieved. Increasing Recall by raising Top‑K often harms Precision, creating a natural tension.

To balance them, practitioners commonly use the harmonic mean F1 Score. However, Precision and Recall ignore ranking quality. In RAG, the order matters because earlier chunks occupy more context tokens. Mean Reciprocal Rank (MRR) evaluates the position of the first relevant chunk (score = 1/rank). Normalized Discounted Cumulative Gain (NDCG) assesses the entire ranked list, weighting higher‑ranked documents more and supporting graded relevance levels.

RAGAS introduces Context Relevance, which estimates the proportion of retrieved context that actually helps answer the question. A typical implementation extracts answer‑useful sentences from the retrieved context with an LLM and computes their ratio to total context.

Generation‑Stage Metrics

After retrieval succeeds, the LLM must produce a high‑quality answer. Faithfulness is the most critical metric: it checks whether the answer stays true to the retrieved documents and does not hallucinate unsupported facts. Evaluation splits the answer into factual claims and verifies each claim against the retrieved context; the score is the fraction of supported claims (used by RAGAS and TruLens).

Answer Relevance measures whether the answer addresses the original question. One method generates reverse questions from the answer with an LLM and computes semantic similarity to the original query.

Answer Completeness/Correctness assesses coverage of all question aspects, typically requiring a ground‑truth reference answer for comparison.

Traditional NLP Metrics

BLEU and ROUGE compute n‑gram overlap with reference texts; they are cheap but ignore semantics, so they serve only as auxiliary signals in RAG. BERTScore improves by using pretrained embeddings to gauge semantic similarity, yet still depends on reference answers and struggles with open‑ended generation.

LLM‑as‑Judge and Automation Frameworks

Human annotation is accurate but costly. The prevailing solution is LLM‑as‑Judge: a strong model (e.g., GPT‑4) automatically scores the RAG outputs on Faithfulness, Answer Relevance, Context Relevance, etc. The RAGAS framework implements this approach, while TruLens offers customizable feedback functions.

LLM‑as‑Judge can make mistakes and incurs inference cost, so a practical workflow combines rapid automated judging for daily iteration with periodic human‑annotated calibration.

End‑to‑End Closed‑Loop Evaluation

While stage‑wise metrics pinpoint problems, the ultimate goal remains end‑to‑end user satisfaction. A robust RAG evaluation stack typically has three layers: fine‑grained stage metrics for quick debugging, automated end‑to‑end scores (e.g., overall RAGAS score) for version comparison, and periodic human/user feedback for calibration. All three layers are complementary and essential.