While absolute positional encodings, as discussed in Chapter 4, provide a mechanism for Transformers to understand sequence order, they treat each position independently. The sinusoidal encoding offers nice properties for modeling relative distances implicitly, but it's added before the attention mechanism operates. Learned absolute embeddings might struggle to generalize to sequence lengths longer than those seen during training. An alternative approach directly incorporates the relative distance between tokens into the attention calculation itself. This is the core idea behind Relative Positional Encodings (RPE).

The intuition is that the relationship between two words often depends more on how far apart they are rather than their absolute positions in the sequence. For instance, knowing that a verb follows its subject by one position might be a more generalizable pattern than knowing the subject is at position 5 and the verb is at position 6. RPE aims to make the model directly aware of these relative distances.

Modifying the Attention Score

Instead of adding positional information to the input embeddings, RPE modifies the self-attention scoring mechanism. The standard scaled dot-product attention calculates scores between a query $q_i$ for position $i$ and a key $k_j$ for position $j$ as:

\text{score}(q_i, k_j) = \frac{q_i^T k_j}{\sqrt{d_k}}

Where $q_i = x_i W^Q$ and $k_j = x_j W^K$ , with $x_i, x_j$ being input embeddings and $W^Q, W^K$ being projection matrices.

Relative positioning schemes inject information about the relationship between $i$ and $j$ directly into this calculation. Several variations exist, but they generally involve adding terms that depend on the relative distance $i-j$ .

Shaw et al. (2018) Formulation

One of the earlier influential approaches proposed adding learned relative position embeddings directly to the keys (and sometimes values) before the dot product. Let $a_{ij}^K$ and $a_{ij}^V$ represent learnable embedding vectors corresponding to the relative position between query $i$ and key/value $j$ . The attention score calculation is modified as:

e_{ij} = \frac{(x_i W^Q)^T (x_j W^K + a_{ij}^K)}{\sqrt{d_k}}

The output value $z_i$ is then computed using a similar modification for the value vectors:

z_i = \sum_j \text{softmax}(e_{ij}) (x_j W^V + a_{ij}^V)

Here, $a_{ij}^K$ and $a_{ij}^V$ are typically retrieved from embedding lookup tables indexed by the relative distance $j-i$ . To keep the number of embeddings manageable, the relative distance is often clipped to a maximum value $k$ . That is, all distances $j-i > k$ map to the same embedding $a_{i, i+k}^K$ , and distances $j-i < -k$ map to $a_{i, i-k}^K$ .

This approach directly injects relative spatial biases into the attention score. However, it requires computing and storing these relative embeddings for each query-key pair within the attention matrix computation, which can be computationally intensive.

Transformer-XL / Dai et al. (2019) Formulation

A more efficient and widely adopted approach, introduced with Transformer-XL, reformulates the attention calculation to elegantly incorporate relative positions. Recall the standard attention score involving absolute positional embeddings $P_i, P_j$ :

A_{i,j}^{\text{abs}} = (E_{x_i} + P_i)^T W^{Q T} W^K (E_{x_j} + P_j)

Expanding this gives four terms: content-content ( $E_{x_i}^T \dots E_{x_j}$ ), content-position ( $E_{x_i}^T \dots P_j$ ), position-content ( $P_i^T \dots E_{x_j}$ ), and position-position ( $P_i^T \dots P_j$ ).

The relative formulation modifies this expansion:

Replace absolute position $P_j$ in the key projection: In terms involving the key's position, replace the absolute position $P_j$ with a relative position encoding $R_{i-j}$ that represents the offset between $i$ and $j$ . $R$ can be a fixed sinusoidal encoding matrix (similar to the original Transformer's PE, but used differently) or learned embeddings.
Introduce trainable position biases: Replace the query's absolute position term $P_i^T W^{Q T}$ with two trainable vectors, $u$ and $v$ . These vectors represent global "positional biases" for content and relative position, respectively.

This leads to the following decomposed attention score calculation:

A_{i,j}^{\text{rel}} = \underbrace{E_{x_i}^T W^{Q T} W^K E_{x_j}}_{\text{(a) content-based}} + \underbrace{E_{x_i}^T W^{Q T} W^K R_{i-j}}_{\text{(b) content-relative position}} + \underbrace{u^T W^K E_{x_j}}_{\text{(c) global content bias}} + \underbrace{v^T W^K R_{i-j}}_{\text{(d) global positional bias}}

Term (a) is identical to the content interaction in standard attention.
Term (b) captures how the query content at $i$ relates to the relative position $i-j$ .
Term (c) provides a bias based purely on the key content at $j$ .
Term (d) provides a bias based purely on the relative position $i-j$ .

Crucially, this formulation can be implemented efficiently. The terms involving $R_{i-j}$ can be computed without explicitly constructing pairwise relative embeddings for all $(i, j)$ pairs. Instead, clever tensor manipulations allow computing terms (b) and (d) efficiently across all positions simultaneously.

Implementation Aspects

Clipping Distance: As with the Shaw et al. approach, a maximum relative distance $k$ is often used when indexing relative position embeddings ( $R_{i-j}$ ). This assumes that very long-range interactions might not need precise distance information.
Embedding Type: The relative position representation $R$ can be based on sinusoidal functions (providing generalization capabilities) or learned embeddings (potentially more expressive but requiring more data and parameters).
Sharing: Relative position embeddings (whether sinusoidal or learned) are often shared across different attention heads and sometimes across layers to reduce the parameter count.

Advantages of Relative Positional Encoding

Improved Generalization: RPEs, especially sinusoidal ones or those with clipping, may generalize better to sequence lengths not seen during training compared to learned absolute PEs. The model learns patterns based on distance rather than specific locations.
Direct Distance Modeling: The attention mechanism becomes directly aware of the relative positioning between tokens, which can be beneficial for tasks where local syntax or relative order is important.
Empirical Success: RPEs form a component of several high-performing models, including Transformer-XL, T5, and DeBERTa, demonstrating their practical effectiveness.

Disadvantages of Relative Positional Encoding

Increased Complexity: The attention calculation becomes more involved compared to the baseline Transformer, although efficient implementations like the Transformer-XL formulation mitigate the computational overhead significantly compared to naive approaches.
Hyperparameters: Introduces choices like the clipping distance $k$ and the type of relative encoding (sinusoidal vs. learned), which may require tuning.

In summary, relative positional encodings offer a compelling alternative to absolute positional encodings by embedding sequence order information directly into the attention mechanism's score calculation. By focusing on pairwise distances rather than absolute locations, they can provide better generalization and capture distance-sensitive relationships more effectively, proving valuable in various modern Transformer architectures.