Shared Representations: Learning Common Features

The chapter introduction explained that we need ways to bring information from different modalities together. We've looked at fusion strategies, where data streams are merged. Now, let's explore another powerful approach: learning shared representations.

Imagine you're trying to get two people who speak different languages to understand each other. You could translate word-for-word (which is a bit like some fusion methods), or you could try to get them both to understand a set of common ideas or symbols. Shared representations in multimodal AI are more like the latter.

The main idea is to transform data from different modalities, like an image and a piece of text, into a common format or "space" where they can be directly compared. Think of it as creating a universal language that both images and text (or audio, etc.) can be translated into. If an image and a sentence describe the same thing, their translations into this universal language should be very similar.

What is a Shared Representation?

A shared representation, sometimes called a joint embedding space, is a vector space where representations (or "embeddings") from two or more different modalities can coexist and be meaningfully compared.

Let's say we have an image, $x_{\text{img}}$, and a text caption, $x_{\text{txt}}$, that describes the image. The goal is to learn two mapping functions, $f_{\text{img}}(\cdot)$ and $f_{\text{txt}}(\cdot)$, such that: $z_{\text{img}} = f_{\text{img}}(x_{\text{img}})$ $z_{\text{txt}} = f_{\text{txt}}(x_{\text{txt}})$ Here, $z_{\text{img}}$ and $z_{\text{txt}}$ are vectors in the same high-dimensional space (e.g., both might be 300-dimensional vectors). The "shared" part means that if $x_{\text{img}}$ and $x_{\text{txt}}$ are semantically related (e.g., the image depicts what the text describes), then their corresponding vectors $z_{\text{img}}$ and $z_{\text{txt}}$ should be "close" to each other in this space.

Why Learn Shared Representations?

Creating these common spaces is useful for several reasons:

1. Cross-Modal Comparison and Similarity: Once data from different modalities are in the same space, you can directly measure how similar they are. For example, you can calculate the distance (e.g., Euclidean distance) or similarity (e.g., cosine similarity) between an image vector and a text vector. A small distance or high similarity would suggest they are related.
2. Cross-Modal Retrieval: This is a very common application.
   • Text-to-Image Retrieval: Given a text query (e.g., "a dog catching a frisbee"), you can convert it to a vector in the shared space and then search for images whose vectors are closest to the text query's vector.
   • Image-to-Text Retrieval: Conversely, given an image, you can find text descriptions that best match it.
3. Data Alignment: Shared spaces can help align data from different sources. For instance, aligning spoken words in an audio track with the corresponding text transcript.
4. Information Transfer: Sometimes, learning a good representation for one modality can help improve the representation of another, especially if there's a strong correlation between them.

How are Shared Representations Learned?

Typically, shared representations are learned using neural networks. Each modality will often have its own network (or "encoder") responsible for processing its specific type of data and transforming it into an initial feature vector. Then, these feature vectors are projected into the shared embedding space.

The magic happens during the training process. The model is trained on pairs (or triplets) of data from different modalities. For instance, it might be given:

• An image and its correct caption (a positive pair).
• The same image and an incorrect caption (a negative pair).

The learning algorithm, guided by a loss function, tries to adjust the network parameters so that:

• The vectors for positive pairs (e.g., $z_{\text{img}}$ for the image and $z_{\text{txt}}$ for its correct caption) are pulled closer together in the shared space.
• The vectors for negative pairs (e.g., $z_{\text{img}}$ for the image and $z_{\text{txt\_wrong}}$ for an unrelated caption) are pushed further apart.

This process, often involving techniques like contrastive learning or triplet loss (though the specifics are beyond our current scope), effectively "shapes" the shared space so that semantic similarity across modalities translates into proximity in the vector space.

Here's a diagram illustrating the general idea for images and text:

Data from different modalities (image, text) are processed by their respective encoders and mapped into a shared semantic space where their vector representations can be directly compared.

For example, if you have an image of a "blue car parked by a red fire hydrant" and the text "a blue car next to a red hydrant," their representations $z_{\text{img}}$ and $z_{\text{txt}}$ in the shared space would ideally be very close. If the text was "a cat sleeping on a mat," its vector $z_{\text{txt\_other}}$ should be far from $z_{\text{img}}$.

Benefits and Challenges

Shared representation learning is a powerful technique for many multimodal tasks, especially those involving search, retrieval, and comparison.

Benefits:

• Direct Cross-Modal Interaction: Enables direct comparison and relation of information from different sources.
• Flexibility: The learned representations can often be used for various downstream tasks.
• Improved Understanding: By forcing modalities into a common space, models can learn deeper semantic relationships.

Challenges:

• Defining Similarity: Determining what "semantically similar" means and encoding that into a loss function can be difficult.
• Space Alignment: Ensuring that the different modalities are properly aligned in the shared space and that the space itself is well-structured (e.g., not collapsing all points to one region) is a significant technical challenge.
• Scalability: Training these models often requires large amounts of paired multimodal data.

While methods like early or late fusion, which we discussed previously, directly combine features or decisions, shared representation learning focuses on creating a common ground for understanding. This approach provides a different, and often very effective, way to integrate information from multiple modalities. Next, we'll look at another related idea: coordinated representations.