Building upon the concept of attention introduced previously, this chapter focuses on the specific attention mechanisms used within Transformer models. We will examine self-attention, a technique that allows the model to weigh the significance of different words within the same input sequence when processing a specific word.

You will learn how input embeddings are projected into Query ($Q$), Key ($K$), and Value ($V$) vectors, which form the basis for calculating attention scores. We will detail the Scaled Dot-Product Attention formula:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

where $d_k$ is the dimension of the key vectors.

Furthermore, we will examine Multi-Head Attention. This approach involves running the Scaled Dot-Product Attention mechanism multiple times in parallel with different, learned linear projections of $Q$, $K$, and $V$. This allows the model to jointly attend to information from different representation subspaces at different positions. We will cover its mechanics and the reasoning behind its effectiveness. Finally, the chapter includes a practical exercise where you will implement the Scaled Dot-Product Attention mechanism using a deep learning library.