Context Vectors from Attention Weights

Attention mechanisms calculate scores, essentially assigning a relevance weight ($\alpha$) to each input element based on a query. These scores indicate how much attention to pay to different parts of the input sequence. These scores are then used to create a context vector.

Think of the context vector as a distilled summary of the input sequence, tailored specifically to the current focus or query. Instead of just compressing the entire input into a single, fixed-size vector like traditional RNNs often do, attention allows us to create a dynamic summary that emphasizes the most relevant parts.

From Scores to Summary: The Weighted Sum

The core idea is remarkably straightforward: the context vector is calculated as a weighted sum of the Value vectors ($v$) from the input sequence. The weights used in this sum are precisely the attention scores ($\alpha$) we calculated previously.

Mathematically, if we have $n$ input elements, each with a corresponding Value vector $v_i$ and an attention score $\alpha_i$ (relative to the current query), the context vector $C$ is computed as:

$$C = \sum_{i=1}^{n} \alpha_i v_i$$

Let's break this down:

1. Value Vectors ($v_i$): Remember, these vectors represent the content or meaning of each input element.
2. Attention Scores ($\alpha_i$): These are the non-negative weights calculated in the previous step (often via softmax), summing up to 1 ($\sum_{i=1}^{n} \alpha_i = 1$). Each $\alpha_i$ reflects the importance of input element $i$ for the current query.
3. Weighted Sum: We multiply each Value vector $v_i$ by its corresponding attention score $\alpha_i$. This scales the contribution of each value vector based on its calculated relevance. Value vectors corresponding to highly relevant inputs (high $\alpha_i$) will contribute more significantly to the sum, while less relevant inputs (low $\alpha_i$) will have their influence diminished.
4. Context Vector ($C$): The resulting vector $C$ is the sum of these weighted Value vectors. It has the same dimension as the Value vectors but represents a blended combination of the inputs, weighted by relevance.

Visualizing the Process

Imagine you have three input words, each represented by a Value vector ($v_1, v_2, v_3$), and you've calculated attention weights ($\alpha_1, \alpha_2, \alpha_3$) for them based on a specific query. The context vector $C$ is formed by blending these vectors according to the weights.

The diagram shows how each Value vector ($v_i$) is multiplied by its corresponding attention weight ($\alpha_i$). These weighted vectors are then summed together to produce the final context vector ($C$).

Why is this Useful?

This mechanism is powerful because it allows the model to dynamically create a relevant summary of the input for each step of the output generation (or for each element being processed).

For example, in machine translation, when translating a sentence, the meaning of a specific output word often depends heavily on one or two words in the input sentence, plus some broader context. A traditional RNN might struggle to retain precise information from distant input words due to its sequential nature and fixed-size hidden state.

Attention, however, calculates weights for all input words at each output step. The resulting context vector directly incorporates information from the most relevant input words, regardless of their position in the sequence. This overcomes the information bottleneck of fixed-size context vectors and allows models to handle long-range dependencies much more effectively.

This context vector, enriched with focused information from the input, becomes a primary input for the next stage of processing, such as generating the next word in a sequence or feeding into subsequent layers in the network. We'll see exactly how this fits into the larger Transformer architecture in the upcoming chapters.