Component-Level Evaluation: Retrieval

Evaluating the performance of the retriever component is essential because the quality of the final generated answer heavily depends on the relevance and accuracy of the information fetched from your knowledge base. If the retriever provides irrelevant, outdated, or incorrect context documents (or document chunks) to the Large Language Model (LLM), the generator has little chance of producing a useful response. This follows the classic "garbage in, garbage out" principle. The core question we need to answer is: Given a user query, how well does the retriever identify and rank the most pertinent information from the indexed data?

Establishing Ground Truth

To quantitatively measure retriever performance, you typically need a benchmark or "ground truth" dataset. This usually consists of:

A set of representative queries ( $Q$ ).
For each query $q \in Q$ , a set of known relevant document chunks ( $D_q$ ) from your knowledge base.

Creating this ground truth often involves manual annotation by subject matter experts who identify which specific chunks contain the necessary information to answer each query. While time-consuming, this step is fundamental for calculating objective performance metrics. Without it, evaluation becomes purely qualitative or relies on indirect measures.

Common Retrieval Metrics

Assuming you have a ground truth dataset, several standard Information Retrieval (IR) metrics can be adapted to evaluate your RAG retriever. These metrics typically assess the relevance and ranking of the documents returned for a given query, usually focusing on the top $K$ results, as RAG systems often use only the highest-ranked chunks due to context window limitations.

Hit Rate

Hit Rate is one of the simplest metrics. It measures whether any relevant document chunk was retrieved within the top $K$ results for a given query. For a single query, the Hit Rate is 1 if at least one relevant chunk is found in the top $K$ retrieved chunks, and 0 otherwise. The overall Hit Rate is the average across all queries in your test set.

Calculation: For each query, check if $D_{retrieved}@K \cap D_{q} \neq \emptyset$ . Average the results (1s and 0s).
Interpretation: A higher Hit Rate indicates the retriever is generally capable of finding some relevant information.
Limitation: It's a binary metric per query and doesn't care where the relevant document appears within the top $K$ . Finding a relevant document at rank $K$ scores the same as finding it at rank 1.

Mean Reciprocal Rank (MRR)

Mean Reciprocal Rank (MRR) addresses the ranking limitation of Hit Rate. It focuses on the rank of the first relevant document retrieved. For a single query, the Reciprocal Rank is $\frac{1}{\text{rank}}$ , where 'rank' is the position of the highest-ranked relevant document. If no relevant document is found in the top $K$ results, the reciprocal rank for that query is typically 0. MRR is the average of these reciprocal ranks across all queries.

Formula: $MRR = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{\text{rank}_i}$ where $|Q|$ is the total number of queries, and $\text{rank}_i$ is the rank of the first relevant document for the $i$ -th query.
Interpretation: MRR rewards retrievers that place relevant documents higher in the results list. A perfect score of 1 means the first relevant document was always ranked first. Scores closer to 0 indicate relevant documents are ranked lower or not found.
Example:
- Query 1: First relevant doc at rank 2 -> RR = 1/2
- Query 2: First relevant doc at rank 1 -> RR = 1/1
- Query 3: No relevant doc in top K -> RR = 0
- MRR = (1/2 + 1/1 + 0) / 3 = (0.5 + 1 + 0) / 3 = 1.5 / 3 = 0.5

Precision@K

Precision at K (Precision@K) measures the proportion of retrieved documents in the top $K$ results that are actually relevant.

Formula: $\text{Precision@K} = \frac{|\{\text{Relevant documents}\} \cap \{\text{Retrieved documents in top K}\}|}{K}$
Interpretation: High Precision@K means that a large fraction of the documents shown to the LLM are relevant. This is important for avoiding the "context stuffing" problem, where irrelevant information might distract the LLM or occupy valuable context window space.
Example: If $K=5$ and the retriever returns 3 relevant documents and 2 irrelevant documents in the top 5, Precision@5 = 3/5 = 0.6.

Recall@K

Recall at K (Recall@K) measures the proportion of all relevant documents (in the ground truth for that query) that were successfully retrieved within the top $K$ results.

Formula: $\text{Recall@K} = \frac{|\{\text{Relevant documents}\} \cap \{\text{Retrieved documents in top K}\}|}{|\{\text{Total relevant documents for query}\}|}$
Interpretation: High Recall@K indicates that the retriever is successful at finding most of the necessary information for a query within the top $K$ results. This is often critical for RAG, as missing a single essential piece of information can lead to an incomplete or incorrect answer.
Example: If there are 4 ground truth relevant documents for a query, and the retriever finds 3 of them in the top $K=5$ results, Recall@5 = 3/4 = 0.75.

Comparison of MRR scores for two different retriever configurations based on chunking strategy. Configuration B shows improved performance.

Qualitative Analysis

While quantitative metrics provide valuable scores, they don't tell the whole story. It's also important to perform qualitative analysis:

Examine Top Results: For a sample of queries (especially those with low metric scores), manually inspect the top $K$ retrieved chunks. Are they truly relevant? Are they repetitive? Do they contain conflicting information?
Analyze Missed Relevance: If Recall@K is low, investigate why known relevant chunks were not retrieved or ranked highly. Is it an issue with the embedding model's understanding, the query formulation, or the document chunking?
Identify Irrelevant Results: If Precision@K is low, look at the irrelevant results that are being retrieved. Are they semantically similar but contextually wrong? This can point to ambiguities that the embedding model struggles with or suggest improvements needed in the source data or chunking strategy.

Evaluating the retriever component using a combination of quantitative metrics (like Hit Rate, MRR, Precision@K, Recall@K) and qualitative analysis provides a comprehensive understanding of its strengths and weaknesses. This allows you to make informed decisions about tuning parameters (like $K$ , chunk size, embedding model) or refining your data preparation process to improve the quality of context provided to the generator.

Build LLM apps faster with Kerb

Cleaner syntax. Built-in debugging. Production-ready from day one.

Built for the AI systems behind ApX Machine Learning

Was this section helpful?

References

Introduction to Information Retrieval, Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze, 2008 (Cambridge University Press) - A comprehensive textbook covering fundamental concepts and metrics in information retrieval, including precision, recall, and Mean Reciprocal Rank (MRR), which are essential for retriever evaluation.
A Survey of Retrieval-Augmented Generation for Large Language Models, Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, Haofen Wang, 2024 arXiv preprint arXiv:2312.10997 DOI: 10.48550/arXiv.2312.10997 - Provides a broad overview of Retrieval-Augmented Generation (RAG) systems, including sections that discuss evaluation methodologies and challenges for different RAG components, such as the retriever.
BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, Iryna Gurevych, 2021 NeurIPS 2021 Dataset and Benchmark Track DOI: 10.48550/arXiv.2104.08663 - Introduces a widely recognized benchmark dataset and framework for evaluating information retrieval models, offering practical guidance for creating ground truth and assessing retriever performance.