The standard Transformer architecture, while effective, presents computational challenges, primarily the quadratic complexity $O(N^2)$ associated with self-attention relative to the input sequence length $N$. This complexity limits the practical application of Transformers to very long sequences.

This chapter examines these limitations and introduces several architectural modifications designed to enhance efficiency and performance. We will analyze the computational cost of vanilla self-attention and then investigate alternatives including:

• Sparse Attention Mechanisms: Techniques that restrict the attention computation to specific patterns, reducing the number of query-key comparisons.
• Linear Attention Approximations: Methods like Linformer and Performer that approximate the attention matrix using techniques such as low-rank projections or kernel methods, aiming for linear $O(N)$ complexity.
• Transformer-XL: Incorporating recurrence to handle longer contexts more effectively.
• Relative Positional Encodings: An alternative way to represent sequence order based on pairwise distances.
• Normalization Placement: Comparing Pre-LN and Post-LN variants and their impact on training dynamics.
• Scaling Laws: Empirical observations regarding the relationship between model size, dataset size, compute, and performance.
• Parameter Efficiency: Techniques for reducing the number of model parameters without sacrificing significant performance.

By studying these variants, you will gain insight into the ongoing research and development aimed at making Transformer models more scalable and efficient for diverse applications.