Transformers rely entirely on attention mechanisms, lacking the inherent sequence awareness of RNNs. Therefore, injecting positional information is essential. While the standard sinusoidal and learned absolute positional encodings introduced earlier provide a baseline, they possess certain limitations, particularly regarding generalization to longer sequences and explicitly capturing relative distances between tokens.

This chapter examines alternative approaches designed to encode positional information more effectively or efficiently. We will cover:

• The motivations for moving beyond absolute positional encodings.
• Concepts behind relative positional encoding, where the relationship between positions is encoded.
• Implementations such as adding relative position biases directly to attention scores (Shaw et al.).
• The relative encoding scheme used in Transformer-XL.
• Rotary Position Embedding (RoPE), a method that applies rotations to query and key vectors based on position.

By the end of this chapter, you will understand the mechanics of these advanced positional encoding techniques and the contexts where they offer advantages over standard absolute encodings.