Meta-learning fundamentally shifts the learning objective from mastering a single task to acquiring the ability to learn new tasks rapidly and efficiently. Unlike traditional supervised learning where a model is trained on a large dataset for one specific job, meta-learning operates on a distribution of tasks. The objective is to extract transferable knowledge or learning strategies that accelerate adaptation when faced with a novel task, particularly when data for that new task is scarce. This is often framed as "learning to learn."

The Meta-Learning Setup: Tasks, Support, and Query Sets

At the heart of the meta-learning problem lies the concept of a task. A task $T$ represents a specific learning problem, such as classifying a new set of images, translating between a novel pair of languages, or adapting a language model to a unique writing style. In the meta-learning framework, we assume tasks are drawn from an underlying probability distribution $p(T)$ . This distribution defines the universe of problems the meta-learning algorithm is expected to handle.

The meta-learning process typically involves two phases:

Meta-Training: The algorithm is exposed to a collection of tasks, $D_{meta-train} = \{T_1, T_2, ..., T_N\}$ , sampled from $p(T)$ . The goal during this phase is not to master any individual task $T_i$ perfectly, but rather to learn a general model initialization, a learning procedure, or a metric space that facilitates fast learning on future tasks drawn from the same distribution.
Meta-Testing: The learned model or procedure is evaluated on a separate set of new tasks, $D_{meta-test} = \{T'_1, T'_2, ..., T'_M\}$ , also sampled from $p(T)$ , which were not seen during meta-training. Performance on these tasks measures the algorithm's ability to generalize its learned learning strategy.

Anatomy of a Meta-Learning Task

Crucially, each individual task $T_i$ within the meta-training or meta-testing set is itself structured as a small learning problem. It comprises two distinct subsets of data:

Support Set ( $S_i$ ): This is a small set of labeled examples specific to task $T_i$ . Its purpose is to provide the data needed for the model to adapt or learn the specifics of this particular task during the meta-learning process (often within an "inner loop"). For a K-shot, N-way classification task, the support set typically contains $K$ examples for each of the $N$ classes. Formally, $S_i = \{(x_{i,k}, y_{i,k})\}_{k=1}^{|S_i|}$ . The size of $S_i$ is intentionally kept small, reflecting the few-shot learning scenario.
Query Set ( $Q_i$ ): This is another set of labeled examples from the same task $T_i$ , disjoint from the support set ( $S_i \cap Q_i = \emptyset$ ). Its purpose is to evaluate the model's performance after it has adapted using the support set $S_i$ . The loss calculated on the query set often drives the optimization of the meta-parameters (the "outer loop" update). Formally, $Q_i = \{(x'_{i,j}, y'_{i,j})\}_{j=1}^{|Q_i|}$ .

This division into support and query sets within each task is fundamental. It simulates the real-world few-shot scenario during meta-training: the model must learn from the support set how to perform well on the query set for that specific task.

Data flow within a single step of the meta-training phase. A task $T_i$ is sampled, split into support ( $S_i$ ) and query ( $Q_i$ ) sets. The meta-model parameters ( $\theta$ ) are adapted using $S_i$ to yield task-specific parameters ( $\phi_i$ ). Performance is then evaluated on $Q_i$ , and the resulting loss informs the update to the meta-parameters $\theta$ .

The Meta-Objective

Let $\theta$ represent the parameters of our meta-learned model or the parameters defining our learning procedure (e.g., initial weights of a neural network, parameters of an optimizer). The process of adapting these general parameters $\theta$ to task-specific parameters $\phi_i$ using the support set $S_i$ can be denoted by a function or algorithm $\text{Adapt}$ . So, $\phi_i = \text{Adapt}(\theta, S_i)$ .

The ultimate goal of meta-training is to find the optimal meta-parameters $\theta^*$ that minimize the expected loss on the query sets $Q_i$ across the distribution of tasks $p(T)$ , after adaptation using the corresponding support sets $S_i$ . If $L(Q_i, \phi_i)$ represents the loss (e.g., cross-entropy, mean squared error) of the adapted model $\phi_i$ on the query set $Q_i$ , the meta-objective can be formally stated as:

\theta^* = \arg\min_{\theta} \mathbb{E}_{T_i \sim p(T)} [ L(Q_i, \phi_i) ]

\theta^* = \arg\min_{\theta} \mathbb{E}_{T_i \sim p(T)} [ L(Q_i, \text{Adapt}(\theta, S_i)) ]

In practice, this expectation is approximated by averaging the query set loss over a batch of tasks sampled during each meta-training iteration.

Relevance to Foundation Models

This formulation directly applies when working with large foundation models. Here, $\theta$ represents the potentially massive set of parameters of the foundation model (e.g., a Transformer). The Adapt function could be:

Fine-tuning (all or part of $\theta$ ) on $S_i$ .
Learning parameter-efficient modules (like adapters or LoRA matrices) using $S_i$ , while keeping the base $\theta$ frozen.
A non-parametric method (like k-Nearest Neighbors on embeddings derived from $\theta$ ) using $S_i$ .
A gradient-based meta-learning algorithm (like MAML) performing simulated adaptation steps on $S_i$ .

Regardless of the specific Adapt mechanism, the meta-learning goal remains consistent: find initial parameters $\theta$ (or a way to generate them) such that the model performs well on the query set $Q_i$ after seeing only the small support set $S_i$ . The challenge lies in efficiently performing the meta-optimization (finding $\theta^*$ ) and the adaptation ( $\text{Adapt}(\theta, S_i)$ ) given the enormous scale of $\theta$ in foundation models, a central theme explored throughout this course. Understanding this core problem structure is essential before examining specific algorithms designed to solve it.