Few-Shot Learning with CNNs

Standard transfer learning, involving fine-tuning a pre-trained model, typically assumes a reasonable amount of labeled data for the target task. However, in many practical scenarios, acquiring large labeled datasets is infeasible due to cost, time, or the inherent rarity of certain categories. Imagine needing to identify a newly discovered species of bird from only a handful of photographs or adapting a medical imaging system to recognize a rare condition with just a few patient examples. This is where Few-Shot Learning (FSL) becomes essential.

FSL tackles the problem of learning to recognize new classes given only a very small number of labeled examples, often just one or five per class. Formally, this is often framed as an $N$-way $K$-shot classification problem: the model is given $K$ labeled examples (the "support set") for each of $N$ new classes it hasn't seen during initial training, and its goal is to correctly classify new, unlabeled examples (the "query set") belonging to one of these $N$ classes. When $K=1$, it's called one-shot learning.

Directly fine-tuning a large CNN like ResNet or EfficientNet with only $K$ examples per class often leads to severe overfitting. The model's high capacity allows it to simply memorize the few support examples without learning generalizable features for the new classes. Therefore, specialized techniques are required. FSL methods generally fall into a few categories, often building upon strong feature representations learned via pre-training on a larger, related dataset (like ImageNet).

Metric Learning Approaches

The core idea behind metric learning for FSL is to learn an embedding function that maps images into a feature space where images from the same class are close together, and images from different classes are far apart. Classification can then be performed by comparing the embedding of a query image to the embeddings of the support examples.

Prototypical Networks

Prototypical Networks offer an intuitive and effective metric learning approach. During training and testing, they operate on "episodes" designed to mimic the few-shot scenario.

1. Embedding: A CNN backbone (often pre-trained) processes both support and query images to extract feature embeddings. Let $f_\phi$ be the embedding function parameterized by $\phi$.
2. Prototype Calculation: For each of the $N$ classes in the support set $S = \{(x_1, y_1), ..., (x_{NK}, y_{NK})\}$, where $x_i$ is an image and $y_i$ is its class label, a single prototype vector $c_n$ is computed. This is typically the mean of the embeddings of the $K$ support examples belonging to class $n$: $$c_n = \frac{1}{K} \sum_{(x_i, y_i) \in S_n} f_\phi(x_i)$$ where $S_n$ is the subset of support examples for class $n$.
3. Classification: A query image $x_q$ is classified based on its distance to the class prototypes in the embedding space. A distribution over classes is computed using a softmax over the negative distances (or similarities). Using squared Euclidean distance $d$, the probability of $x_q$ belonging to class $n$ is: $$p(y=n | x_q) = \frac{\exp(-d(f_\phi(x_q), c_n))}{\sum_{n'=1}^N \exp(-d(f_\phi(x_q), c_{n'}))}$$
4. Training: The network parameters $\phi$ are learned by minimizing the negative log-likelihood of the true class for the query examples across many sampled episodes.

The episodic training forces the embedding function $f_\phi$ to produce representations that generalize well to new classes, as it must consistently form clusters and separate prototypes for classes it hasn't been specifically trained on before each episode.

Siamese Networks

Another foundational metric learning technique involves Siamese Networks. These networks process pairs of images using identical CNNs (sharing weights $\phi$). The network outputs embeddings for both images, and a distance function (like Euclidean distance or cosine similarity) compares these embeddings. The network is trained to minimize the distance for pairs of images from the same class and maximize it for pairs from different classes, often using a contrastive loss or triplet loss function. For few-shot classification, a query image's embedding can be compared against the embeddings of all support images, and classification is typically done based on the nearest support example's class.

Optimization-Based Approaches (Meta-Learning)

Instead of learning a fixed embedding space, optimization-based methods focus on learning an algorithm or model initialization that can quickly adapt to new tasks using only a few examples. This is often referred to as "learning to learn" or meta-learning.

Model-Agnostic Meta-Learning (MAML)

MAML is a popular and versatile meta-learning algorithm. Its goal is to find a set of initial model parameters $\theta$ such that adapting these parameters to a new task requires only a few gradient steps using the task's small support set, leading to good performance on that task's query set.

The process involves two optimization loops:

1. Inner Loop (Task Adaptation): For a specific sampled task (an episode in FSL context), the model parameters $\theta$ are temporarily updated for one or a few gradient steps using the support set $S_{task}$. This results in task-specific parameters $\theta'_{task}$. For a one-step update with learning rate $\alpha$: $$\theta'_{task} = \theta - \alpha \nabla_\theta \mathcal{L}_{S_{task}}(\theta)$$ where $\mathcal{L}_{S_{task}}$ is the loss computed on the support set of the current task.
2. Outer Loop (Meta-Optimization): The initial parameters $\theta$ are updated based on the performance of the adapted parameters $\theta'_{task}$ on the query set $Q_{task}$ of the same task. The loss is computed using the adapted parameters, but the gradients are taken with respect to the original parameters $\theta$. $$\theta \leftarrow \theta - \beta \nabla_\theta \sum_{task \sim p(task)} \mathcal{L}_{Q_{task}}(\theta'_{task})$$ where $\beta$ is the meta-learning rate.

This outer loop update involves differentiating through the inner loop's gradient update, often requiring second-order derivatives (though first-order approximations are common). MAML aims to find an initialization $\theta$ that is positioned strategically in the parameter space, making it highly sensitive and adaptable to various few-shot tasks drawn from the task distribution $p(task)$.

Episodic Training Explained

Many FSL methods, particularly metric-learning approaches like Prototypical Networks, rely heavily on episodic training. This strategy directly simulates the few-shot problem during the training phase.

The episodic training process samples small $N$-way $K$-shot tasks (episodes) from a larger base dataset. The model is trained to perform well on the query examples within each episode, based only on the support examples provided for that episode.

In each training iteration:

1. $N$ classes are randomly selected from the base training dataset.
2. $K$ support examples are randomly sampled for each of the $N$ classes.
3. $Q$ query examples are randomly sampled from the remaining examples of those same $N$ classes.
4. The model processes the support set (e.g., computes prototypes).
5. The model makes predictions on the query set.
6. A loss is calculated based on the query set predictions.
7. Model parameters are updated via backpropagation.

By repeatedly training on these diverse, randomly generated few-shot tasks, the model learns representations or adaptation strategies that are effective even for entirely new classes it encounters during meta-testing, provided they come from a similar distribution.

Connection to Transfer Learning

Few-shot learning is intrinsically linked to transfer learning. Most successful FSL approaches do not train the entire CNN from scratch using only the episodic procedure. Instead, they often use a CNN backbone pre-trained on a large dataset (like ImageNet) as a powerful feature extractor. The FSL technique (metric learning, MAML, etc.) is then applied primarily to the final layers or operates within the feature space produced by this pre-trained backbone. The pre-training provides a strong foundation of general visual features, which the FSL method then adapts or utilizes specifically for the task of discriminating between new classes based on few examples.

Practical Notes

• Base Dataset: The choice and size of the base dataset used for pre-training or for sampling episodes significantly impacts performance. A base dataset relevant to the target few-shot tasks is generally beneficial.
• Backbone Architecture: The capacity and suitability of the CNN backbone architecture (e.g., ResNet variations) are important.
• Evaluation: FSL models are evaluated on separate meta-testing episodes drawn from classes completely unseen during training. Performance is typically averaged over many such test episodes.
• Method Choice: Metric learning methods like Prototypical Networks are often simpler to implement and computationally less demanding than optimization-based methods like MAML, but the best choice can be task-dependent.

Few-shot learning provides a powerful set of tools for adapting CNNs in data-scarce environments, extending the reach of computer vision to applications where massive labeled datasets are not available. It represents a sophisticated form of model adaptation, improving learning under significant data constraints.