How to Evaluate RAG Systems: Key Metrics and the Ragas Framework

In practical deployments, evaluating RAG (Retrieval‑Augmented Generation) projects is essential. The author uses the Ragas automated evaluation framework, which splits RAG assessment into four dimensions: Recall Quality (whether the system retrieves correct and relevant document fragments), Answer Faithfulness (whether the large model avoids fabricating answers), Answer Relevance (whether the model’s response actually addresses the user’s question), and Context Utilization (how much of the provided context the model actually uses).

Answer Correctness

This metric checks if the answer is correct by comparing the model’s response with a reference answer, verifying that facts, conclusions, and key points match. The core is simply whether the result is right.

Answer Relevancy

This metric evaluates whether the answer is on‑topic. Even if the content is factually correct, a low score is given if the response does not directly answer the user’s question, deviates, or is overly generic. The core is whether the answer hits the user’s real intent.

Faithfulness

This metric assesses whether the answer is grounded in the supplied context, i.e., whether the content can be traced back to the retrieved material rather than being hallucinated by the model. The core is the presence of hallucinations and whether the answer is supported by context.

Context Precision

Context Precision measures the proportion of truly useful content in the retrieved results. For example, if 10 retrieved passages contain only 2 relevant ones, the precision score is low; a higher proportion of relevant passages yields a higher score. The core is the accuracy of retrieval and the amount of noise.

Context Recall

Context Recall evaluates how comprehensively the retrieval stage gathers the information needed to answer the question. If answering requires three key pieces of evidence but only one is retrieved, recall is low. The core is the completeness of retrieval and whether critical material is missed.

The article also includes two illustrative diagrams showing the relationship between retrieval‑stage metrics (Context Precision, Context Recall) and generation‑stage metrics (Answer Correctness, Answer Relevancy, Faithfulness).