Building Ground Truth Datasets for Evaluation

To quantitatively assess the performance of your vector search system using metrics like Recall@k or Precision@k, you first need a standard against which to compare the system's output. This standard is the ground truth: a collection of queries paired with the set of documents or items considered truly relevant for each query. Thorough evaluation, which relies on such a standard, is a primary step for tuning parameters and ensuring your system meets its objectives. Building a high-quality ground truth dataset is often the most challenging, yet most significant, part of this evaluation process.

The Challenge of Defining Relevance

Before constructing a ground truth dataset, you must grapple with the inherent subjectivity of "relevance". What one user considers relevant, another might not. Relevance is highly dependent on the query's intent, the user's context, and the specific task (e.g., finding a single precise answer for RAG versus exploring related concepts in semantic search).

Challenges include:

1. Subjectivity: Defining objective criteria for relevance that can be consistently applied is difficult.
2. Scope: How many items are truly relevant for a given query? Is it a fixed number, or all items exceeding a certain relevance threshold?
3. Graded Relevance: Often, relevance isn't binary (relevant/not relevant). Some documents might be highly relevant, others only marginally so. Capturing these nuances requires a more complex labeling scheme.
4. Dynamic Data: If your underlying document corpus changes frequently, the ground truth may become outdated.

Despite these challenges, establishing a working definition of relevance and building a corresponding ground truth dataset is essential for systematic evaluation.

Methods for Constructing Ground Truth Datasets

"Several approaches exist for building ground truth, each with its own trade-offs regarding cost, scalability, and fidelity to relevance."

Human Labeling and Annotation

This is often considered the most reliable method, as it directly involves human judgment to assess relevance.

• Process:

  1. Query Selection: Choose a representative set of queries that reflect expected user behavior, covering frequent (head), moderately common (torso), and rare (tail) queries.
  2. Candidate Generation: For each query, retrieve a pool of potential candidate documents. This pool should be larger than the expected number of relevant items and might come from multiple sources (e.g., existing search methods, random sampling) to reduce bias from a single retrieval system.
  3. Guideline Definition: Create clear, specific annotation guidelines defining what constitutes relevance for the task. Include examples of different relevance levels if using a graded scale (e.g., 0 = Not Relevant, 1 = Marginally Relevant, 2 = Relevant, 3 = Highly Relevant).
  4. Annotation: Have multiple trained human annotators independently rate the relevance of each candidate document for each query based on the guidelines.
  5. Adjudication: Resolve disagreements between annotators, often through discussion or a senior reviewer.

• Inter-Annotator Agreement (IAA): Measuring the consistency between annotators is important for assessing the quality of the labels and the clarity of the guidelines. Common metrics include Cohen's Kappa (for two annotators) or Krippendorff's Alpha (for multiple annotators). Low IAA scores often indicate ambiguous guidelines or subjective tasks.

• Tooling: This process can be managed using spreadsheets, custom-built annotation tools, or dedicated platforms like Amazon SageMaker Ground Truth, Labelbox, or open-source options like doccano.

While providing high-quality labels, human annotation is typically expensive, slow, and requires careful management.

Using Implicit Signals from User Behavior

Instead of direct human judgment, you can infer relevance from user interactions logged by your application.

• Signals: Common signals include clicks, add-to-carts, purchases, session duration, copy-pasting content, or saving items. The assumption is that users interact more with relevant results.
• Process: Aggregate interaction data for query-document pairs. Pairs with high positive interaction rates (e.g., high click-through rate for a given query) can be designated as relevant.
• Challenges:
  • Noise: Clicks can be misleading (accidental clicks, curiosity clicks on irrelevant items). Users might not click on relevant items presented lower down.
  • Sparsity: Many query-document pairs will have little to no interaction data, especially for tail queries or new documents.
  • Presentation Bias: Results shown higher up are naturally clicked more often, regardless of absolute relevance.

Implicit signals are scalable and reflect real user behavior but require careful cleaning, filtering, and interpretation. They are often better suited for relative comparisons (e.g., A/B testing) than for establishing absolute ground truth for metrics like Recall.

Synthetic Ground Truth Generation

This involves programmatically creating query-document pairs where relevance is assumed based on the generation process.

• Methods:
  • Question-Answer Pairs: Use existing FAQs or have an LLM generate questions based on document passages. The document becomes the relevant ground truth for the generated question.
  • Duplicate/Paraphrase Detection: Use datasets where documents are known paraphrases or near-duplicates. One document can serve as the query for the other.
  • Document Structure: For structured documents, use titles or section headings as queries and the corresponding content as the relevant document.
• Pros: Can be generated quickly and cheaply at scale. Useful for bootstrapping evaluation.
• Cons: Generated queries may not reflect natural user language or intent. May not capture the full complexity of relevance.

"Synthetic data is useful for initial system checks or when other methods are infeasible, but should be validated against performance."

Leveraging Canonical Benchmark Datasets

For common tasks like web search or question answering, standardized datasets exist.

• Examples: MS MARCO (passage ranking), TREC datasets (various information retrieval tracks), SQuAD (question answering), Natural Questions (QA), BEIR (benchmark suite for diverse retrieval tasks).
• Pros: Allow comparison against published results, well-defined evaluation protocols.
• Cons: May not match your specific data domain, distribution, or definition of relevance. Using them requires aligning your data format and embedding strategy with the benchmark.

Using benchmarks is valuable for academic comparison or validating components but may not fully represent the performance within your specific application context.

Structuring and Maintaining Ground Truth

Once collected, ground truth data needs to be structured for use in evaluation scripts. A common format maps query identifiers to a list of relevant document identifiers, potentially with relevance scores.

Example JSON structure:

[
  {
    "query_id": "q_001",
    "query_text": "how to tune hnsw parameters",
    "relevant_docs": [
      {"doc_id": "doc_abc", "relevance": 3},
      {"doc_id": "doc_xyz", "relevance": 2},
      {"doc_id": "doc_pqr", "relevance": 3}
    ]
  },
  {
    "query_id": "q_002",
    "query_text": "scalar quantization vs product quantization",
    "relevant_docs": [
      {"doc_id": "doc_lmn", "relevance": 3},
      {"doc_id": "doc_def", "relevance": 1}
    ]
  }
]

• Query Selection: Ensure the queries used for ground truth creation cover a range of topics, lengths, and expected difficulty relevant to your application.
• Dataset Size: The number of queries needed depends on the desired statistical significance and diversity of the task. Start with a smaller set (e.g., 50-100 queries) and expand as needed.
• Bias Awareness: Be mindful of biases introduced during query selection (e.g., only expert queries) or annotation (e.g., groupthink among annotators).
• Maintenance: Periodically review and potentially update the ground truth dataset, especially if the underlying document collection changes significantly or user needs evolve.

Using Ground Truth for Metric Calculation

This carefully constructed dataset forms the basis for calculating relevance metrics. For instance, to calculate Recall@k for a specific query:

1. Retrieve the top $k$ results from your vector search system for that query.
2. Look up the list of known relevant document IDs for that query in your ground truth dataset.
3. Count how many of the system's top $k$ results are present in the ground truth list.
4. Divide this count by the total number of relevant documents listed in the ground truth for that query (or sometimes capped at $k$, depending on the precise definition).

Let $R_q$ be the set of relevant document IDs for query $q$ from the ground truth, and $S_{q,k}$ be the set of the top $k$ document IDs returned by the search system for query $q$. Then:

$$\text{Recall}@k(q) = \frac{|S_{q,k} \cap R_q|}{|R_q|}$$

Similarly, Precision@k, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (nDCG) rely on comparing system output against the ground truth, often incorporating graded relevance scores if available.

Building a ground truth dataset is a non-trivial but indispensable investment. It provides the objective basis needed to measure progress, compare different algorithms or parameter settings, and ultimately ensure your vector search system effectively meets the information needs of your users.