Previous chapters detailed the self-attention mechanism, which allows models to weigh the significance of different input elements relative to each other. However, the core attention calculation, operating on sets of queries, keys, and values, does not inherently account for the sequential order of the input. This chapter addresses this gap.

We will first look at the input embedding layer, responsible for converting input tokens into continuous vector representations. The primary focus will then shift to techniques for encoding positional information. We will study the widely used sinusoidal positional encoding scheme, including its mathematical formulation, such as:

$PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d_{model}})$ $PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d_{model}})$

We will analyze the properties that make these functions suitable and how they are typically combined with the token embeddings. We will also discuss alternative approaches like learned positional embeddings and compare their characteristics.