Prototypical Networks Revisited

Prototypical Networks offer an elegant approach within metric-based meta-learning, centered on the idea of creating a representative prototype for each class within a task's support set. Classification of a query point then relies on finding the nearest class prototype in a learned embedding space, $f_\phi(\cdot)$. While the fundamental concept is straightforward, its effective application, particularly with high-dimensional embeddings from foundation models, requires a closer examination of its core components.

The Embedding Function $f_\phi(\cdot)$

The success of Prototypical Networks is critically dependent on the quality of the embedding function $f_\phi$. This function maps input data points $x$ to vectors $z = f_\phi(x)$ in an embedding space $\mathbb{R}^D$. The goal is for this space to exhibit a structure where points from the same class cluster together, while points from different classes are well-separated.

In the context of foundation models (like large vision transformers or language models), $f_\phi$ is often derived directly from the pre-trained model's intermediate or final layer representations. This uses the rich, general-purpose features learned during large-scale pre-training. Key points to examine include:

1. Choice of Layer: Embeddings from different layers capture varying levels of abstraction. Earlier layers might retain more fine-grained details, while deeper layers capture more semantic information. The optimal choice is often task-dependent and requires empirical validation.
2. Frozen vs. Fine-Tuned Embedder:
   • Frozen: Using the foundation model purely as a fixed feature extractor ($\phi$ is not updated during meta-training) is computationally efficient. Its success relies on the assumption that the pre-trained space is already suitable for few-shot classification via prototypes.
   • Fine-Tuned: Updating $\phi$ during meta-training allows the embedding space to adapt specifically for prototypical classification across various tasks. This can lead to better performance but significantly increases computational cost and requires careful optimization, especially given the scale of foundation models. Techniques like parameter-efficient fine-tuning (PEFT), discussed in Chapter 5, might be applied to $\phi$ during meta-training.

Prototype Computation

For a given task $\mathcal{T}$ and a class $k$, the prototype $c_k$ is typically computed as the mean vector of the embedded support examples $\{x_i, y_i\}_{i=1}^{N_S}$ belonging to that class:

$$c_k = \frac{1}{|S_k|} \sum_{(x_i, y_i) \in S_k} f_\phi(x_i)$$

where $S_k = \{(x_i, y_i) \in S \mid y_i = k\}$ is the subset of the support set $S$ belonging to class $k$.

Flow of Prototypical Networks: Support set examples are embedded, averaged to form class prototypes. The embedded query point is classified based on its distance to these prototypes.

This mean aggregation acts as a simple form of noise reduction and summarization. However, in extremely low-shot scenarios (e.g., 1-shot learning), the prototype is simply the embedding of the single support example, making it sensitive to outliers or atypical examples. While alternative aggregation methods exist (e.g., weighted means, robust estimators), the standard mean remains prevalent due to its simplicity and effectiveness when the embedding space is well-structured.

Distance Metrics

The choice of distance function $d(\cdot, \cdot)$ used to compare the embedded query point $z_q = f_\phi(x_q)$ with the class prototypes $c_k$ is another important design decision.

1. Squared Euclidean Distance ($L_2^2$): This is the most common choice, often used in the original formulation. The probability of the query point $x_q$ belonging to class $k$ is modeled via a softmax over the negative distances: $$p(y=k | x_q) = \frac{\exp(-d(z_q, c_k))}{\sum_{k'} \exp(-d(z_q, c_{k'}))} = \frac{\exp(-\|f_\phi(x_q) - c_k\|^2_2)}{\sum_{k'} \exp(-\|f_\phi(x_q) - c_{k'}\|^2_2)}$$
2. Cosine Distance: In high-dimensional spaces, which are typical for foundation model embeddings ($D \gg 1000$), Euclidean distance can be problematic due to the concentration of measure phenomenon, where distances between points tend to become less discriminative. Cosine similarity, or its corresponding distance ($d_{cos}(u, v) = 1 - \frac{u \cdot v}{\|u\| \|v\|}$), focuses on the angle between vectors rather than their magnitude. This can be more robust in high dimensions and is often preferred when dealing with embeddings where vector magnitude might not carry significant class-specific information or varies greatly. Using cosine distance implicitly involves L2 normalization of the embeddings and prototypes.

The choice between Euclidean and cosine distance often depends on the properties of the embedding space generated by $f_\phi$ and whether vector magnitudes are meaningful for discrimination.

Inductive Bias

Prototypical Networks carry a strong inductive bias: they assume that each class can be adequately represented by a single point (the prototype) in the embedding space and that classes form roughly hyperspherical clusters around these prototypes when using Euclidean distance. This works well if the embedding function $f_\phi$ is trained (or inherently structured, in the case of pre-trained models) to map classes into such well-separated, compact regions.

When using embeddings from foundation models, this bias might not always align perfectly with the data's structure for a specific downstream few-shot task, especially if the task involves classes with significant intra-class variance or multi-modal distributions. The effectiveness depends on whether the general-purpose representations learned by the foundation model happen to cluster classes well, or whether meta-training $f_\phi$ successfully imposes this structure.

Connection to Deep Metric Learning

Training the embedding function $f_\phi$ is effectively a deep metric learning problem. The goal is to learn a transformation that brings samples from the same class closer together and pushes samples from different classes further apart, specifically in a way that supports prototype-based classification. While Prototypical Networks define the classification mechanism (mean prototypes + distance), the training of $f_\phi$ during meta-learning often implicitly or explicitly optimizes metric learning objectives. For instance, the loss function derived from the softmax over distances to prototypes encourages the necessary cluster structure. More explicit metric learning losses, such as triplet loss (discussed in Section 3.4), can also be incorporated into the meta-training process to shape the embedding space more directly.

Revisiting Prototypical Networks reveals that their simplicity belies important underlying choices regarding the embedding function, prototype calculation, and distance metric. These choices become particularly significant when scaling to the high-dimensional representations typical of foundation models, requiring careful examination of computational constraints, inductive biases, and the geometric properties of the embedding space.