Query, Key, and Value Vectors in Self-Attention

Self-attention aims for each word in an input sequence to determine the relevance of every other word (including itself) within that specific context. To achieve this comparison and subsequent information weighting, each input word's embedding is transformed into three distinct representations: the Query, Key, and Value vectors.

Consider an input sequence like "thinking machine". Each word is initially represented by an embedding vector. Let's denote the embedding for "thinking" as $x_1$ and for "machine" as $x_2$. These vectors hold the initial, context-independent meaning of the words.

For the self-attention calculation, we don't directly compare these embeddings. Instead, we project each embedding $x_i$ into three separate vector spaces using three unique weight matrices learned during training. These are commonly denoted as $W^Q$, $W^K$, and $W^V$.

Specifically, for every input embedding $x_i$ in the sequence, we compute:

1. A Query vector: $q_i = x_i W^Q$
2. A main vector: $k_i = x_i W^K$
3. A Value vector: $v_i = x_i W^V$

This process generates a unique set of query, and value vectors for each word in the input sequence. These derived vectors typically have a dimension ($d_k$ for keys and queries, $d_v$ for values) that might be smaller than the original embedding dimension ($d_{model}$). The requirement is that the dimensions of Query and Key vectors ($d_k$) must be identical to allow for their comparison via dot products.

We can visualize this transformation for a single input token:

Derivation of Query (q), (k), and Value (v) vectors from a single input embedding (xᵢ) using learned weight matrices (W^Q, W^K, W^V).

To understand the roles of these vectors, consider an analogy of searching a database:

• Query (Q): Represents the current word's perspective, asking "What information is relevant to me?". For a specific word $i$, its query vector $q_i$ is used to probe all other words.
• Key (K): Represents the 'label' or identifier for the information a word carries. Each word $j$ has a key vector $k_j$. The query $q_i$ is compared against all keys $k_j$ to calculate an attention score, indicating how relevant word $j$ is to word $i$.
• Value (V): Represents the actual content or meaning of a word. Each word $j$ also has a value vector $v_j$. Once the attention score between $q_i$ and $k_j$ is computed, it's used to weight the corresponding value vector $v_j$.

Essentially, the interaction between a Query vector $q_i$ and a Key vector $k_j$ determines the strength of connection or attention weight from word $i$ to word $j$. The Value vector $v_j$ provides the information that gets passed from word $j$ back to word $i$, scaled by this attention weight.

An important aspect is that the weight matrices $W^Q$, $W^K$, and $W^V$ are parameters learned during the model training process. Initially random, they are adjusted through backpropagation so the model learns the most effective way to project input embeddings into these Q, K, and V spaces for the task at hand (e.g., machine translation, text summarization). This learning process allows the model to understand complex relationships and dependencies within the input sequence.

Having generated these Query, Key, and Value vectors for every word, we are now equipped to calculate the actual attention scores. The next section details how the Scaled Dot-Product Attention mechanism utilizes these vectors to compute the precise attention weights that define how information flows between words in the sequence.