Building upon the scaled dot-product attention mechanism, this chapter focuses on self-attention, a specific application where a sequence relates different positions within itself to compute its representation. Applying only a single attention function, however, can limit its ability to capture diverse relational aspects simultaneously, potentially forcing an undesirable averaging effect.

To overcome this limitation, we examine Multi-Head Attention. This approach allows the model to jointly attend to information from different representation subspaces at different positions, effectively running the attention mechanism in parallel multiple times.

This chapter covers:

• The definition of self-attention where Queries ($Q$), Keys ($K$), and Values ($V$) derive from the same input sequence.
• The motivation for using multiple parallel attention layers, known as 'heads'.
• The mechanics of projecting $Q$, $K$, and $V$ using distinct linear transformations for each head.
• How attention is computed in parallel across heads.
• The process of concatenating head outputs and applying a final linear projection.
• How different heads can potentially learn distinct relational patterns.
• A practical guide to implementing a multi-head attention module.

By understanding these components, you gain insight into a fundamental building block of the Transformer architecture.