The Need for Positional Information

The self-attention mechanism provides a powerful way for a model to dynamically weigh the importance of different elements within an input sequence when computing a representation for each element. It achieves this by calculating alignment scores between pairs of elements using Query, Key, and Value vectors derived from the input.

However, a significant characteristic of the core self-attention calculation, specifically the scaled dot-product attention:

Attention(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

is that it is inherently permutation invariant. This means that if we were to reorder the input tokens (and their corresponding $Q, K, V$ vectors), the attention output for any given token, considering its relationship to all other tokens, would effectively remain the same, just reordered according to the initial permutation. The attention mechanism, in its basic form, treats the input as a set of vectors rather than an ordered sequence.

Consider a simple sentence: "robot detects anomaly". If the model processes this using only token embeddings and self-attention, the attention scores between "robot" and "detects", or "robot" and "anomaly", are computed based solely on the vector representations of these words, irrespective of their positions (1st, 2nd, or 3rd). If we shuffled the input to "anomaly detects robot", the pairwise attention scores between the embeddings would be identical. The resulting contextualized representations would be different only because the input vectors themselves are different at each position, but the mechanism itself hasn't processed the order.

This contrasts sharply with recurrent architectures like RNNs or LSTMs. In an RNN, the computation at time step $t$ directly depends on the hidden state from time step $t-1$ : $h_t = f(h_{t-1}, x_t)$ . This sequential processing naturally incorporates the order of elements. Transformers, by processing all tokens in parallel via self-attention, lose this built-in sense of sequence order.

For almost any task involving sequences, especially in natural language processing, order is fundamental to meaning. "The dog chased the cat" has a completely different meaning from "The cat chased the dog". Time series predictions rely heavily on the temporal ordering of observations. Therefore, since the self-attention mechanism itself doesn't capture positional information, we must find an alternative way to provide this information to the model. The representations fed into the Transformer layers need to contain signals that indicate not just what a token is, but also where it is located within the sequence. This is the primary motivation for incorporating positional encodings, which we will explore next.

Was this section helpful?

References

Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, 2017 Advances in Neural Information Processing Systems, Vol. 30 (Curran Associates, Inc.) DOI: 10.5555/3295222.3295289 - Introduces the Transformer architecture, including the scaled dot-product attention and the necessity for positional encodings due to its permutation invariance.
Natural Language Processing with Transformers, Lewis Tunstall, Leandro von Werra, Thomas Wolf, 2022 (O'Reilly Media) - Offers an accessible yet comprehensive explanation of Transformer models, detailing the architecture's components, including the need for and implementation of positional information.
Deep Sets, Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, Alexander J Smola, 2017 Advances in Neural Information Processing Systems (NeurIPS), Vol. 30 (Curran Associates, Inc.) DOI: 10.5555/3295222.3295328 - A foundational paper that mathematically characterizes permutation-invariant neural networks, offering a theoretical context for understanding why basic self-attention treats input as a set.