ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Advanced Transformer Architecture
Chapter 1: Revisiting Sequence Modeling Limitations
Sequential Computation in Recurrent Networks
The Vanishing and Exploding Gradient Problems
Long Short-Term Memory (LSTM) Gating Mechanisms
Gated Recurrent Units (GRUs) Architecture
Challenges with Long-Range Dependencies
Parallelization Constraints in Recurrent Models
Chapter 2: The Attention Mechanism: Core Concepts
Motivation: Overcoming Fixed-Length Context Vectors
General Framework: Query, Value Abstraction
Mathematical Formulation of Dot-Product Attention
Scaled Dot-Product Attention
The Softmax Function for Attention Weights
Computational Aspects and Matrix Operations
Practice: Implementing Scaled Dot-Product Attention
Chapter 3: Multi-Head Self-Attention
Self-Attention: Queries, Keys, Values from the Same Source
Limitations of Single Attention Head
Introducing Multiple Attention Heads
Linear Projections for Q, K, V per Head
Parallel Attention Computations
Concatenation and Final Linear Projection
Analysis of What Different Heads Learn
Hands-on Practical: Building a Multi-Head Attention Layer
Chapter 4: Positional Encoding and Embedding Layer
The Need for Positional Information
Input Embedding Layer Transformation
Sinusoidal Positional Encoding: Formulation
Properties of Sinusoidal Encodings
Combining Embeddings and Positional Encodings
Alternative: Learned Positional Embeddings
Comparison: Sinusoidal vs. Learned Embeddings
Practice: Generating and Visualizing Positional Encodings
Chapter 5: Encoder and Decoder Stacks
Overall Transformer Architecture Overview
Encoder Layer Structure
Decoder Layer Structure
Masked Self-Attention in Decoders
Encoder-Decoder Cross-Attention
Position-wise Feed-Forward Networks (FFN)
Residual Connections (Add)
Layer Normalization (Norm)
Stacking Multiple Layers
Final Linear Layer and Softmax Output
Hands-on Practical: Constructing an Encoder Block
Chapter 6: Advanced Architectural Variants and Analysis
Computational Complexity of Self-Attention
Sparse Attention Mechanisms
Approximating Attention: Linear Transformers
Kernel-Based Attention Approximation (Performers)
Low-Rank Projection Methods (Linformer)
Transformer-XL: Segment-Level Recurrence
Relative Positional Encodings
Pre-Normalization vs Post-Normalization (Pre-LN vs Post-LN)
Scaling Laws for Neural Language Models
Parameter Efficiency and Sharing Techniques
Chapter 7: Implementation Details and Optimization
Choosing a Framework (PyTorch, TensorFlow, JAX)
Weight Initialization Strategies
Optimizers for Transformers (Adam, AdamW)
Learning Rate Scheduling (Warmup, Decay)
Regularization Techniques (Dropout, Label Smoothing)
Gradient Clipping
Mixed-Precision Training
Efficient Attention Implementations (FlashAttention)
Model Parallelism and Data Parallelism Strategies
Practice: Analyzing Attention Weight Distributions

Encoder-Decoder Cross-Attention

While the masked self-attention mechanism allows the decoder to attend to previous tokens within the sequence it's generating, it lacks direct access to the information encoded from the input sequence. To bridge this gap and enable the decoder to selectively focus on relevant parts of the source information, the Transformer architecture incorporates a second attention mechanism within each decoder layer: encoder-decoder cross-attention.

Unlike self-attention where Queries (Q), Keys (K), and Values (V) all originate from the same sequence (either the input sequence in the encoder or the partially generated output sequence in the decoder's masked self-attention), cross-attention draws its components from different sources:

  1. Queries (Q): These vectors come from the output of the preceding sub-layer in the decoder. Specifically, they are derived from the output of the masked self-attention layer after the residual connection and layer normalization step. These queries represent the decoder's current state and what information it needs to generate the next token.
  2. Keys (K) and Values (V): These vectors come directly from the output of the final layer of the encoder stack. The encoder has processed the entire input sequence, and its output representation contains contextualized information about the source. The keys and values derived from this representation allow the decoder to "look back" at the input sequence.

This structure allows the decoder, at each step, to pose a query based on its current context (the generated sequence so far) and match it against the keys representing the entire input sequence. The resulting attention scores determine how much weight to give to the corresponding values from the encoder's output when constructing the decoder's output for that step.

Mathematical Formulation

The computation itself uses the same scaled dot-product attention function discussed previously. However, the inputs clearly distinguish the sources:

Attention(Qdec,Kenc,Venc)=softmax(QdecKencTdk)Venc\text{Attention}(Q_{dec}, K_{enc}, V_{enc}) = \text{softmax}\left(\frac{Q_{dec} K_{enc}^T}{\sqrt{d_k}}\right) V_{enc}Attention(Qdec​,Kenc​,Venc​)=softmax(dk​​Qdec​KencT​​)Venc​

Here:

  • QdecQ_{dec}Qdec​ represents the query matrices derived from the decoder's previous sub-layer output.
  • KencK_{enc}Kenc​ and VencV_{enc}Venc​ represent the matrices derived from the encoder stack's output.
  • dkd_kdk​ is the dimension of the vectors, used for scaling.

The softmax function ensures the weights assigned to the encoder's value vectors sum to one, creating a weighted average based on relevance as determined by the query-key interactions.

Multi-Head Cross-Attention

Just like self-attention, cross-attention benefits from multiple attention heads operating in parallel. Each head applies separate linear projections to the incoming QdecQ_{dec}Qdec​, KencK_{enc}Kenc​, and VencV_{enc}Venc​ to map them into different representation subspaces.

headi=Attention(QdecWiQ,KencWiK,VencWiV)\text{head}_i = \text{Attention}(Q_{dec}W_i^Q, K_{enc}W_i^K, V_{enc}W_i^V)headi​=Attention(Qdec​WiQ​,Kenc​WiK​,Venc​WiV​)

Where WiQW_i^QWiQ​, WiKW_i^KWiK​, and WiVW_i^VWiV​ are the learned projection matrices for head iii.

The outputs of these independent heads are then concatenated and passed through a final linear projection layer, identical to the process in multi-head self-attention:

MultiHead(Qdec,Kenc,Venc)=Concat(head1,...,headh)WO\text{MultiHead}(Q_{dec}, K_{enc}, V_{enc}) = \text{Concat}(\text{head}_1, ..., \text{head}_h) W^OMultiHead(Qdec​,Kenc​,Venc​)=Concat(head1​,...,headh​)WO

Using multiple heads allows the decoder to simultaneously attend to information from the encoder output based on different criteria or from different representational perspectives. For instance, in translation, one head might focus on syntactic alignment while another focuses on semantic correspondence.

Purpose and Integration

The primary purpose of encoder-decoder cross-attention is to condition the decoder's generation process on the relevant parts of the input sequence. Without it, the decoder would only have access to the input information through the initial decoder state, lacking the ability to dynamically focus on specific source tokens as generation progresses.

For example, translating "The black cat" (input) to "Le chat noir" (output).

  1. When generating "Le", the decoder's masked self-attention looks at the start-of-sequence token. Its cross-attention might weakly attend to the whole input or focus on "The".
  2. When generating "chat", the decoder's masked self-attention looks at "Le". Its cross-attention mechanism, using a query influenced by "Le", should strongly attend to the encoder's representation of "cat".
  3. When generating "noir", the masked self-attention looks at "Le chat". The cross-attention should now strongly attend to the encoder's representation of "black".

This dynamic focusing is fundamental to the Transformer's success in sequence-to-sequence tasks.

Data Flow within the Decoder Layer

The cross-attention mechanism sits between the masked self-attention sub-layer and the position-wise feed-forward sub-layer within each decoder layer. Residual connections and layer normalization are applied around it, just as with the other sub-layers.

Simplified data flow for the Encoder-Decoder Cross-Attention sub-layer within a Transformer Decoder layer. Queries originate from the decoder's state, while Keys and Values come from the encoder's final output.

Understanding this cross-attention mechanism is important for grasping how the Transformer decoder effectively utilizes the encoded representation of the input sequence to guide the generation of the output sequence. It acts as the primary bridge connecting the encoder's processing of the source sequence to the decoder's step-by-step generation of the target sequence.

  • Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017 Advances in Neural Information Processing Systems, Vol. 30 DOI: 10.48550/arXiv.1706.03762 - This foundational paper introduced the Transformer architecture and its attention mechanisms, including the detailed design and mathematical formulation of encoder-decoder cross-attention.
  • Speech and Language Processing (3rd ed. draft), Daniel Jurafsky and James H. Martin, 2025 - This comprehensive textbook offers a clear explanation of the Transformer architecture, with a dedicated section on encoder-decoder attention and its role in sequence-to-sequence models.
  • CS224N: Natural Language Processing with Deep Learning - Lecture on Transformers, Diyi Yang, Tatsunori Hashimoto, 2023 - This course provides a pedagogical approach to understanding Transformer components, with lectures and notes detailing the encoder-decoder cross-attention mechanism within the overall architecture.

© 2026 ApX Machine Learning