This chapter focuses on the fundamentals needed to understand the Transformer architecture. We begin by examining the common tasks involving sequential data, such as machine translation or text generation, and the inherent difficulties in processing such information effectively, especially over long sequences.

To establish a baseline, we will briefly review Recurrent Neural Networks (RNNs), a standard approach for handling sequential data. We will then discuss the practical limitations of RNNs, including issues like capturing long-range dependencies and computational bottlenecks.

This discussion sets the stage for introducing the core idea of the attention mechanism. You will learn how attention provides a way for models to selectively focus on relevant parts of the input sequence when producing an output. We will cover the high-level mechanics of how attention scores are calculated using query, key, and value concepts ($Q$, $K$, $V$) and how these scores are used to create weighted context vectors, forming the building blocks for the more advanced mechanisms discussed later in the course.

By the end of this chapter, you will understand:

• The challenges presented by sequence-to-sequence tasks.
• The basic operation and limitations of RNNs.
• The conceptual basis of the attention mechanism.
• The general process of calculating attention scores and context vectors.