Positional Encoding Explained

The core self-attention mechanism within a Transformer doesn't inherently understand the order of tokens in a sequence. If you shuffled the input words, the self-attention output for a specific word (before positional information is added) would change, but the model wouldn't have a built-in way to know that the order itself was different. This is unlike Recurrent Neural Networks (RNNs), which process sequences step-by-step, inherently incorporating order.

To address this, Transformers inject information about the position of each token directly into the input embeddings. This is achieved using Positional Encoding. The idea is to create a unique vector for each position in the sequence and add this vector to the corresponding token's embedding. This combined embedding thus carries both the semantic meaning of the token and its position within the sequence.

While several methods for positional encoding exist, the original Transformer paper introduced a widely used technique based on sine and cosine functions of different frequencies. For a token at position $pos$ in the sequence and dimension $i$ within the embedding vector of size $d_{\text{model}}$, the positional encoding $PE$ is calculated as follows:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$

Let's break this down:

• $pos$: This is the index of the token's position in the sequence (e.g., 0 for the first token, 1 for the second, and so on).
• $i$: This is the index of the dimension within the embedding vector. It ranges from $0$ to $d_{\text{model}}/2 - 1$. Notice that each pair of dimensions $(2i, 2i+1)$ uses the same frequency but different functions (sine and cosine).
• $d_{\text{model}}$: This is the dimensionality of the input embeddings (and also the dimension of the positional encoding vector). A typical value might be 512 or 768.
• $10000$: This value is a hyperparameter chosen by the authors; it determines the range of frequencies used.

The core idea is that the wavelength of the sine/cosine waves varies across the dimensions. For small values of $i$ (the initial dimensions), the wavelength is short (high frequency). For large values of $i$ (the later dimensions), the wavelength becomes very long (low frequency, approaching a constant value for short sequences). Each position $pos$ thus gets a unique combination of sinusoidal values across the $d_{\text{model}}$ dimensions.

The resulting positional encoding vector $PE_{(pos)}$ has the same dimension ($d_{\text{model}}$) as the input embeddings. This positional encoding vector is then simply added element-wise to the token's input embedding:

$$\text{final\_embedding}_{(pos)} = \text{input\_embedding}_{(pos)} + PE_{(pos)}$$

This addition allows the model to learn representations that combine both the token's meaning and its position.

This specific sinusoidal approach has several appealing properties:

1. Uniqueness: It produces a unique encoding vector for each position in the sequence.
2. Determinism: The encodings are fixed and not learned during training, making them predictable and consistent.
3. Relative Position Information: A significant advantage is that the positional encoding for a future position $pos+k$ can be represented as a linear transformation of the encoding for position $pos$. This property makes it easier for the model's attention mechanisms to learn relationships based on relative positions.
4. Bounded Values: The sine and cosine functions naturally keep the encoding values within a range of [-1, 1], which helps stabilize the numerical properties of the embeddings.
5. Generalization: The model might be able to generalize to sequence lengths longer than those encountered during training, although performance can degrade if the difference is substantial.

Here's a visualization of these positional encoding values across different positions and dimensions:

Visualization of positional encoding values. Each row represents a position in the sequence, and each column represents a dimension in the encoding vector. Notice the different frequencies of the sinusoidal patterns across dimensions and the unique vector for each position.

This injection of positional information happens only once, right at the beginning, before the input embeddings are fed into the first layer of the encoder (or decoder) stack. Subsequent layers within the Transformer then process these position-aware embeddings. Without this step, the Transformer would essentially be processing a "bag of words," losing the sequential nature of language or other ordered data.