Combining Embeddings and Positional Encodings

Integrating token embeddings and positional encodings is a fundamental step in preparing input for Transformer models. Token embeddings represent input tokens as dense vectors, while positional encodings generate distinct signals indicating their positions. The self-attention mechanism, a core component of Transformers, processes input elements without inherent knowledge of their sequential order. Consequently, this positional context must be supplied directly within the input representations fed into the Transformer stack.

The standard and remarkably effective method proposed in the original Transformer paper ("Attention Is All You Need") is straightforward: element-wise addition. If $E \in \mathbb{R}^{L \times d_{model}}$ represents the matrix of token embeddings for a sequence of length $L$, and $P \in \mathbb{R}^{L \times d_{model}}$ represents the corresponding positional encodings, the combined input representation $X \in \mathbb{R}^{L \times d_{model}}$ is computed as:

$$X = E + P$$

This means for each token $i$ in the sequence (where $i$ ranges from $0$ to $L-1$), its final input vector $x_i$ is the sum of its token embedding $e_i$ and its positional encoding $p_i$:

$$x_i = e_i + p_i$$

Here, both the token embedding $e_i$ and the positional encoding $p_i$ must have the same dimensionality, $d_{model}$. This consistency in dimension is a fundamental requirement for the addition operation.

Rationale for Addition

Why simple addition? While other composition functions could be conceived, addition offers several advantages:

1. Simplicity and Efficiency: It's computationally inexpensive, adding minimal overhead.
2. Information Preservation: Addition allows the network to potentially access both the semantic information (from $E$) and the positional information (from $P$). Since subsequent layers involve linear transformations (like the Q, K, V projections in multi-head attention), they can theoretically learn to project the combined embedding $X$ in ways that isolate or utilize components related to either $E$ or $P$ as needed.
3. Compatibility with Sinusoidal Encodings: For sinusoidal encodings, addition preserves the relative positional information they are designed to capture. Because the sinusoidal functions have properties related to linear transformations for relative positions (e.g., $PE_{pos+k}$ can be expressed as a linear function of $PE_{pos}$), adding them to token embeddings provides a consistent way for the model to learn about relative distances.

Implementation Workflow

In practice, combining embeddings and positional encodings typically involves these steps:

1. Token Embedding Lookup: Input token IDs are mapped to their corresponding embedding vectors using an embedding layer (often a trainable lookup table). This yields a tensor of shape [batch_size, sequence_length, d_model].
2. Positional Encoding Generation/Lookup:
   • For sinusoidal encodings, these are usually pre-calculated up to a maximum expected sequence length and stored. The relevant encodings for the current sequence length are then selected. The shape is typically [1, sequence_length, d_model] or [sequence_length, d_model] which can be broadcast or sliced.
   • For learned positional embeddings, a separate embedding layer (lookup table) is used, indexed by position IDs (0, 1, 2,...). This also yields a tensor of shape [1, sequence_length, d_model] or similar, containing trainable vectors.
3. Element-wise Addition: The positional encodings are added to the token embeddings. Broadcasting rules often apply here if the positional encoding tensor doesn't explicitly include the batch dimension.

Flow diagram illustrating the combination of token embeddings and positional encodings via element-wise addition to create the input representation for the Transformer layers.

Approaches to Learned Positional Embeddings

If learned positional embeddings are used instead of sinusoidal ones, the combination mechanism remains the same: element-wise addition. The primary difference is that the vectors representing positions $0, 1, 2, \dots$ are now parameters optimized during the training process, rather than being determined by fixed functions. The model learns the positional representations that are most effective for the task, given the data. The addition step still merges these learned positional signals with the semantic token embeddings.

By adding the positional encodings directly to the token embeddings, we create an input representation $X$ that equips the subsequent Transformer layers with both the 'what' (semantics from $E$) and the 'where' (sequence order from $P$) information necessary for sophisticated sequence understanding. This combined representation forms the input to the first layer of the encoder or decoder stack.