Evaluation Protocols for Few-Shot Learning

Evaluating the ability of a model to learn new tasks from limited data is central to meta-learning research. Unlike standard supervised learning evaluation, which typically involves a single large test set, few-shot learning evaluation requires assessing performance across a distribution of new, previously unseen tasks, each with minimal training data. This necessitates specific protocols and benchmarks to ensure thorough and comparable results.

The N-way K-shot Classification Task

The most common framework for evaluating few-shot learning, particularly in classification, is the N-way K-shot task. Here’s the breakdown:

N: Represents the number of distinct classes in the task.
K: Represents the number of labeled examples provided per class for adaptation (learning). These $N \times K$ examples form the support set ( $S$ ).
Query Set (Q): After adapting using the support set, the model is evaluated on a separate set of examples from the same $N$ classes. This is the query set ( $Q$ ), often containing a different number of examples per class than the support set (e.g., 15 query examples per class).

The goal is to maximize accuracy on the query set $Q$ after learning from the support set $S$ . A typical evaluation involves sampling a large number of such $N$ -way $K$ -shot tasks from a designated set of meta-test classes (classes completely held out during meta-training) and averaging the performance.

For instance, a "5-way 1-shot" task means the model must learn to distinguish between 5 classes given only 1 example of each, and then classify new examples from those 5 classes. A "20-way 5-shot" task involves 20 classes with 5 examples each for learning.

Standard Benchmarks

Several benchmark datasets have become standard for evaluating few-shot learning algorithms, facilitating direct comparisons between different methods. While initially focused on vision, similar principles apply to adapting language models or other foundation models.

Omniglot: A dataset of handwritten characters from 50 different alphabets. Often used for 20-way or 50-way, 1-shot or 5-shot tasks. Its large number of classes (1623 total) but relatively few instances per class makes it suitable for learning class representations.
miniImageNet: A subset of ImageNet designed for few-shot learning. It contains 100 classes, each with 600 images. Standard splits typically use 64 classes for meta-training, 16 for meta-validation, and 20 for meta-testing. Commonly evaluated on 5-way 1-shot and 5-way 5-shot tasks.
TieredImageNet: Another ImageNet subset with a hierarchical structure (grouping classes under broader categories). It offers more classes (608 total, split into 351 train, 97 validation, 160 test) than miniImageNet, potentially reducing the chance of meta-test classes being semantically too close to meta-training classes.
Meta-Dataset: A more challenging benchmark combining multiple datasets (ImageNet, Omniglot, Aircraft, Birds, Textures, Quick Draw, Fungi, VGG Flower) with varying characteristics and data availability per task, simulating more realistic, heterogeneous learning scenarios.

While these originated in computer vision, the adaptation evaluation principles extend to NLP and other domains where foundation models are prevalent. For NLP, benchmarks often involve sampling few-shot classification or sequence labeling tasks from collections like GLUE or SuperGLUE, though constructing diverse task distributions remains an active area.

Evaluation Procedure

A strong evaluation protocol involves these steps:

Data Splitting: Divide the available classes into three disjoint sets: meta-training, meta-validation, and meta-testing. This ensures that the tasks encountered during meta-testing use entirely unseen classes.
Meta-Training: Train the meta-learning algorithm using tasks sampled exclusively from the meta-training classes. The meta-validation set is used for hyperparameter tuning and selecting the best meta-model checkpoint.
Meta-Testing: Evaluate the final meta-learned model (or adaptation strategy) on a large number of tasks sampled only from the meta-test classes.
- For each meta-test task:
  - Sample $N$ classes from the meta-test set.
  - Sample $K$ examples per class to form the support set $S_i$ .
  - Sample a distinct set of examples from the same $N$ classes to form the query set $Q_i$ .
  - Adapt the model using $S_i$ .
  - Calculate the accuracy (or another relevant metric) on $Q_i$ .
Reporting: Report the average accuracy across all meta-test tasks, typically along with a 95% confidence interval to account for the variance in task sampling.

General meta-testing procedure: The meta-learned model is evaluated on multiple independent few-shot tasks sampled from held-out classes. Performance is averaged across these tasks.

Approaches for Foundation Models

When evaluating few-shot adaptation of large foundation models:

Computational Cost: Meta-testing can still be computationally intensive if adaptation involves fine-tuning parts of a large model for every task. Efficient evaluation protocols are needed.
Benchmark Relevance: While standard benchmarks are useful, evaluating performance on tasks highly relevant to the foundation model's intended domain (e.g., specific NLP tasks for an LLM, specialized vision tasks for a Vision Transformer) provides more practical insights.
Parameter Efficiency: Evaluation should often consider not just accuracy but also the computational resources (time, memory, FLOPs) required for adaptation, especially when comparing meta-learning to parameter-efficient fine-tuning (PEFT) methods.

Adhering to standardized evaluation protocols is essential for understanding the capabilities and limitations of different meta-learning approaches, particularly as they are applied to the complex challenge of adapting massive foundation models with minimal task-specific data. This ensures progress is measurable and techniques can be reliably compared.

Was this section helpful?

References

Human-level concept learning through probabilistic program induction, Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum, 2015 Science, Vol. 350 (American Association for the Advancement of Science) DOI: 10.1126/science.aab3050 - This foundational paper introduced the Omniglot dataset and demonstrated few-shot learning capabilities, popularizing the concept of learning from very few examples and the N-way K-shot task formulation.
Optimization as a Model for Few-Shot Learning, Sachin Ravi, Hugo Larochelle, 2017 International Conference on Learning Representations (ICLR) - This paper is widely recognized for introducing the miniImageNet benchmark and a meta-learning framework, establishing common evaluation practices for N-way K-shot tasks in the meta-learning community.
Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples, Vincent Triantafillou, Tyler Zhu, Kevin Han, Dian Peng, Andrei Bursuc, Frank Wood, Sergey Levine, Chelsea Finn, 2019 International Conference on Learning Representations (ICLR) - Introduces Meta-Dataset, a challenging benchmark that combines multiple datasets, promoting evaluation robustness across heterogeneous few-shot learning scenarios and pushing beyond single-dataset evaluations.
Generalizing from a Few Examples: A Survey on Few-Shot Learning, Yaqing Wang, Quanming Yao, James T. Kwok, Lionel M. Ni, 2020 ACM Computing Surveys, Vol. 53 (ACM) DOI: 10.1145/3386252 - This comprehensive survey offers an extensive overview of few-shot learning methods, including a detailed discussion of evaluation protocols, benchmark datasets, and open challenges across various domains.