Recap: Pre-trained Language Models and Transformers

Foundational concepts for adapting Large Language Models (LLMs) include pre-trained language models and the Transformer architecture. An overview covers these primary aspects, emphasizing those most relevant to fine-tuning. Familiarity with deep learning, natural language processing (NLP), and basic LLM principles is beneficial.

The Power of Pre-training

Modern LLMs like GPT, Llama, Claude, BERT, and their variants begin their lifecycle not with a specific task in mind, but through a resource-intensive pre-training phase. During this phase, the model is exposed to massive datasets, often encompassing a significant portion of the publicly available internet text and digitized books, potentially terabytes of data and trillions of tokens.

The learning during pre-training is typically self-supervised. Instead of relying on human-generated labels for specific tasks (like sentiment analysis or translation), the model learns from the inherent structure of the language itself. Common self-supervised objectives include:

Next Token Prediction: Given a sequence of text, predict the very next token (word or subword). This is characteristic of autoregressive models like GPT. The loss function encourages the model to assign high probability to the actual next token observed in the training data.
Masked Language Modeling (MLM): Randomly mask out a percentage of tokens in the input sequence and train the model to predict these masked tokens based on the surrounding context. This is the approach used by models like BERT.

Through these objectives, the model is forced to learn intricate statistical patterns in language, including grammar, syntax, semantic relationships between words, common sense reasoning, and even a degree of factual knowledge embedded within the training corpus. The outcome of this phase is a foundation model, a model with broad linguistic understanding and generative capabilities, but not yet specialized for any particular downstream application.

The Transformer Architecture: Enabling Scale and Context

The remarkable success of these pre-trained models is intrinsically linked to the Transformer architecture, introduced in the paper "Attention Is All You Need". This architecture moved away from the recurrent (RNN, LSTM) or convolutional (CNN) approaches previously dominant in sequence modeling.

The core innovation of the Transformer is the self-attention mechanism. This mechanism allows the model, when processing a token at a specific position in the sequence, to dynamically weigh the importance of all other tokens in the sequence (including itself) and draw information from them. It calculates query (Q), key (K), and value (V) vectors for each token and computes attention scores based on the compatibility between queries and keys. This allows the model to capture long-range dependencies and contextual relationships far more effectively than earlier architectures.

A simplified view of a single Transformer block (Decoder style shown, Encoder is similar). Input representations pass through multi-head self-attention and position-wise feed-forward networks, with residual connections and layer normalization applied after each sub-layer.

Main characteristics that make Transformers suitable for large-scale pre-training include:

Contextual Embeddings: Unlike static word embeddings (like Word2Vec), Transformers generate embeddings that depend on the surrounding context. The representation for "bank" differs in "river bank" versus "investment bank".
Parallelization: Computations within a Transformer layer, particularly the self-attention mechanism, can be significantly parallelized across tokens, making training on large datasets and hardware accelerators (GPUs/TPUs) efficient.
Handling Long Dependencies: Self-attention directly relates tokens across the entire sequence length within a layer, overcoming the vanishing gradient issues that limited RNNs in capturing very long-range dependencies.

Standard components like multi-head attention (running self-attention multiple times in parallel with different learned projections), positional encodings (injecting information about token order), layer normalization, and position-wise feed-forward networks work together within stacked layers to create deep, powerful models.

These pre-trained Transformer models represent a significant advancement, possessing an immense amount of generalized knowledge learned from web-scale data. However, this generality is also their limitation for specific applications. Their raw output might be unfocused, factually inconsistent for a specific domain, or not adhere to desired formats or styles. This is precisely where fine-tuning and adaptation techniques become indispensable, allowing us to mold these powerful foundation models to meet specialized requirements, which is the central theme of this course.

References

Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, 2017 Advances in Neural Information Processing Systems (NeurIPS), Vol. 30 DOI: 10.48550/arXiv.1706.03762 - Introduces the Transformer architecture, including the self-attention mechanism, which enabled large-scale pre-training of language models.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, 2018 Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) DOI: 10.48550/arXiv.1810.04805 - Presents the Bidirectional Encoder Representations from Transformers (BERT) model and introduces the Masked Language Modeling objective for pre-training.
Improving Language Understanding by Generative Pre-Training, Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, 2018 arXiv preprint DOI: 10.48550/arXiv.1803.05380 - Describes the GPT-1 model and the generative pre-training approach using next-token prediction, a foundation for autoregressive language models.
CS224N: Natural Language Processing with Deep Learning, Diyi Yang, Tatsunori Hashimoto, 2025 (Stanford University) - An academic course offering extensive lecture materials and assignments on deep learning methods for NLP, including detailed coverage of Transformers and large language models.