ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

Masterclass

How To Build A Large Language Model
Chapter 1: Introduction to Large-Scale Language Modeling
Defining Large Language Models
Historical Context of Sequence Modeling
The Significance of Scale
Computational Challenges Overview
Software and Hardware Ecosystem
Chapter 2: Mathematical Preliminaries for LLMs
Linear Algebra Review: Vectors and Matrices
Calculus Review: Gradients and Optimization
Probability and Statistics Fundamentals
Numerical Stability Considerations
Notation Used Throughout This Course
Chapter 3: Revisiting Sequence Processing Architectures
Fundamentals of Recurrent Neural Networks (RNNs)
Limitations of Simple RNNs
Long Short-Term Memory (LSTM) Networks
Gated Recurrent Units (GRUs)
Sequence-to-Sequence Models with RNNs
Chapter 4: The Transformer Architecture
Overcoming Recurrence with Attention
Scaled Dot-Product Attention
Multi-Head Attention Mechanism
Positional Encoding Techniques
Encoder and Decoder Stacks
The Role of Layer Normalization and Residual Connections
Chapter 5: Tokenization for Large Vocabularies
The Need for Subword Tokenization
Byte Pair Encoding (BPE) Algorithm
WordPiece Tokenization
SentencePiece Implementation
Handling Special Tokens
Vocabulary Size Selection Trade-offs
Chapter 6: Sourcing and Acquiring Massive Text Datasets
Identifying Potential Data Sources
Utilizing Common Crawl Data
Web Scraping Techniques at Scale
Leveraging Open Licensed Datasets
Data Acquisition Legal Considerations
Chapter 7: Data Cleaning and Preprocessing Pipelines
Strategies for Quality Filtering
Text Normalization Methods
Handling Boilerplate and Markup Removal
Near-Duplicate and Exact Duplicate Detection
Language Identification and Filtering
Building Scalable Preprocessing Pipelines
Chapter 8: Building and Managing Large-Scale Datasets
Data Storage Formats (Text, Arrow, Parquet)
Distributed File Systems (HDFS, S3)
Data Indexing for Efficient Retrieval
Dataset Versioning and Reproducibility
Streaming Data Loaders for Training
Chapter 9: Data Sampling Strategies for Training
Importance of Data Mixture Composition
Source Weighting Strategies
Temperature-Based Sampling
Introduction to Curriculum Learning
Data Pacing and Annealing Schedules
Chapter 10: Implementing the Transformer from Scratch
Setting up the Project Environment
Implementing Scaled Dot-Product Attention
Building the Multi-Head Attention Layer
Implementing the Position-wise Feed-Forward Network
Constructing the Encoder and Decoder Layers
Assembling the Full Transformer Model
Chapter 11: Scaling Transformers: Architectural Choices
Scaling Laws for Neural Language Models
Depth vs Width Trade-offs
Choice of Activation Functions (ReLU, GeLU, SwiGLU)
Normalization Layer Placement (Pre-LN vs Post-LN)
Introduction to Sparse Attention Mechanisms
Chapter 12: Initialization Techniques for Deep Networks
The Importance of Proper Initialization
Xavier (Glorot) Initialization
Kaiming (He) Initialization
Initialization in Transformer Components
Small Initialization for Final Layers
Chapter 13: Positional Encoding Variations
Limitations of Absolute Positional Encodings
Relative Positional Encoding Concepts
Implementation of Shaw et al.'s Relative Position
Transformer-XL Relative Positional Encoding
Rotary Position Embedding (RoPE)
Chapter 14: Advanced Architectural Modifications
Parameter-Efficient Fine-Tuning Needs
Adapter Modules for Transformers
Introduction to Mixture-of-Experts (MoE)
Routing Mechanisms in MoE
Load Balancing in MoE Layers
Chapter 15: Distributed Training Strategies
Motivation: Why Distributed Training?
Data Parallelism (DP)
Tensor Parallelism (TP)
Pipeline Parallelism (PP)
Hybrid Approaches (DP+TP, DP+PP, etc.)
Communication Overhead Analysis
Chapter 16: Implementing Distributed Training Frameworks
Overview of Distributed Training Libraries
Introduction to DeepSpeed
Using DeepSpeed ZeRO Optimizations
Introduction to Megatron-LM
Configuring Tensor and Pipeline Parallelism in Megatron-LM
Combining Frameworks and Strategies
Chapter 17: Optimization Algorithms for LLMs
Review of Gradient Descent Variants (SGD, Momentum)
Adaptive Optimizers: Adam and AdamW
Learning Rate Scheduling Strategies
Gradient Clipping Techniques
Choosing Optimizer Hyperparameters (lr, betas, eps, weight_decay)
Chapter 18: Hardware Considerations for LLM Training
GPU Architectures (NVIDIA Ampere, Hopper)
TPU Architectures (Google TPUs)
Memory Requirements (HBM, GPU RAM)
Interconnect Technologies (NVLink, InfiniBand)
Hardware Selection Trade-offs (Cost, Performance, Availability)
Chapter 19: Checkpointing and Fault Tolerance
The Need for Checkpointing in Long Training Runs
Saving Model State (Weights, Optimizer States)
Handling Distributed Checkpointing
Asynchronous vs Synchronous Checkpointing
Checkpoint Frequency and Storage Management
Resuming Training from Checkpoints
Chapter 20: Mixed-Precision Training Techniques
Introduction to Floating-Point Formats (FP32, FP16, BF16)
Benefits of Lower Precision (Speed, Memory)
Challenges with FP16 Training (Range Issues)
Loss Scaling Techniques
Using BF16 (BFloat16) Format
Framework Support for Mixed Precision (AMP)
Chapter 21: Intrinsic Evaluation Metrics
Concept of Language Model Evaluation
Perplexity: Definition and Calculation
Interpreting Perplexity Scores
Bits Per Character/Word
Effect of Tokenization on Perplexity
Chapter 22: Extrinsic Evaluation on Downstream Tasks
Rationale for Downstream Task Evaluation
Common Downstream NLP Tasks
Fine-tuning Procedures for Evaluation
Standard Benchmarks: GLUE and SuperGLUE
Few-Shot and Zero-Shot Evaluation
Developing Custom Evaluation Tasks
Chapter 23: Analyzing Model Behavior
Challenges in Interpreting LLMs
Attention Map Visualization
Probing Internal Representations
Neuron Activation Analysis
Identifying Failure Modes
Chapter 24: Identifying and Mitigating Training Instabilities
Common Symptoms of Instability
Monitoring Training Metrics (Loss, Grad Norm)
Diagnosing Loss Spikes
Debugging Numerical Precision Issues
Stabilization Techniques Revisited (Clipping, LR, Warmup)
Impact of Architectural Choices on Stability
Chapter 25: Fine-tuning for Alignment: Supervised Fine-Tuning (SFT)
Goals of LLM Alignment
Introduction to Supervised Fine-Tuning (SFT)
Creating High-Quality Instruction Datasets
Data Formatting for SFT (Prompts, Completions)
The SFT Training Process and Hyperparameters
Evaluating SFT Models on Alignment Goals
Chapter 26: Reinforcement Learning from Human Feedback (RLHF)
The RLHF Pipeline Overview
Collecting Human Preference Data
Training the Reward Model (RM)
Introduction to Proximal Policy Optimization (PPO)
RL Fine-tuning with PPO
The Role of the KL Divergence Penalty
Challenges and Considerations in RLHF
Alternatives: Direct Preference Optimization (DPO)
Chapter 27: Model Compression Techniques
Motivation for Model Compression
Weight Quantization (INT8, INT4)
Activation Quantization Considerations
Network Pruning (Structured vs Unstructured)
Knowledge Distillation
Evaluating Performance vs Compression Trade-offs
Chapter 28: Efficient Inference Strategies
Challenges in Autoregressive Decoding
Key-Value (KV) Caching
Optimized Attention Implementations (FlashAttention)
Batching Strategies for Throughput
Speculative Decoding
Chapter 29: Serving LLMs at Scale
API Design for LLM Interaction
Model Serving Frameworks (Triton, TorchServe)
Handling Concurrent Requests
Load Balancing Across Model Instances
Monitoring Serving Performance and Cost
Chapter 30: Continuous Training and Model Updates
Motivation for Continuous Improvement
Strategies for Continual Pre-training
Strategies for Continual Fine-Tuning (SFT/RLHF)
Incorporating New Data Sources Safely
Updating Models with Architectural Changes
Versioning, Deployment, and Rollback Strategies

Monitoring Training Metrics (Loss, Grad Norm)

Effective monitoring is the first line of defense against the instabilities that can derail large-scale language model training. Simply launching a multi-day or multi-week training job and hoping for the best is impractical. Instead, engineers need to continuously observe the model's behavior through important metrics. This allows for early detection of problems, facilitating timely intervention before significant computational resources are wasted. Two of the most informative metrics for diagnosing training health are the training loss and the norm of the gradients.

Tracking the Training Loss

The training loss quantifies how well the model is performing on the training data at any given moment. For language models, this is typically a measure like cross-entropy loss, which reflects the model's uncertainty in predicting the next token.

Expected Behavior: In a healthy training run, the loss should generally decrease over time, often rapidly in the early stages and then more slowly as training progresses, eventually plateauing as the model converges. Minor fluctuations are normal, but the overall trend should be downwards.

Signs of Instability:

  • Loss Spikes: Sudden, sharp increases in the loss value. This often indicates that the model encountered a problematic data batch or that the learning rate is too high, causing unstable updates.
  • NaN or Inf Loss: The loss value becomes NaN (Not a Number) or infinity. This is a critical failure, usually caused by numerical overflow (e.g., dividing by zero, taking the logarithm of zero or a negative number) or exploding gradients. Training cannot proceed once a NaN occurs.
  • Divergence: The loss consistently increases over time, indicating the model's predictions are getting worse, not better.
  • Oscillations: The loss fluctuates wildly without a clear downward trend, potentially suggesting issues with learning rate, optimization algorithm stability, or data quality.

Implementation: Logging the loss is straightforward within a standard training loop. You typically calculate the loss after the forward pass and log its value periodically.

import torch
import torch.nn as nn
# Assume model, data_loader, optimizer are defined

# Example logging setup (can use TensorBoard, WandB, etc.)
def log_metric(step, metric_name, value):
    # Replace with your actual logging implementation
    print(f"Step {step}: {metric_name} = {value:.4f}")

global_step = 0
for batch in data_loader:
    optimizer.zero_grad()

    # Assume inputs and targets are obtained from batch
    inputs = batch['input_ids'].to('cuda')
    targets = batch['labels'].to('cuda')

    outputs = model(inputs)
    # Assuming loss calculation involves reshaping for cross_entropy
    loss_fct = nn.CrossEntropyLoss()
    loss = loss_fct(outputs.view(-1, model.config.vocab_size), targets.view(-1))

    # Log the loss (e.g., every 10 steps)
    if global_step % 10 == 0:
        log_metric(global_step, "train_loss", loss.item())

    loss.backward()
    optimizer.step()
    global_step += 1

    # Check for NaN loss early
    if torch.isnan(loss):
        print(f"NaN loss detected at step {global_step}! Stopping training.")
        break

Visualizing the loss curve over time is essential. Tools like TensorBoard or Weights & Biases make this easy.

Example loss curves showing stable training, a sudden loss spike, and divergence where the loss increases over time. Note the logarithmic y-axis, common for viewing loss.

Monitoring the Gradient Norm

The gradient is the vector indicating the direction and magnitude of the steepest ascent of the loss function. The norm of this vector, typically the L2 norm (∣∣∇L∣∣2||\nabla L||_2∣∣∇L∣∣2​), measures its magnitude. Monitoring the gradient norm provides insights into the scale of the updates being applied to the model's weights.

∣∣∇L∣∣2=∑p∈parameters∣∣∇pL∣∣22||\nabla L||_2 = \sqrt{\sum_{p \in \text{parameters}} ||\nabla_p L||_2^2}∣∣∇L∣∣2​=p∈parameters∑​∣∣∇p​L∣∣22​​

Where ∇pL\nabla_p L∇p​L is the gradient of the loss LLL with respect to a specific parameter tensor ppp.

Why it's Important:

  • Exploding Gradients: An excessively large gradient norm indicates that the updates to the weights will be huge, potentially leading to numerical overflow, loss spikes, and divergence. This is a common cause of training instability, especially in deep networks.
  • Vanishing Gradients: A very small or zero gradient norm suggests that the updates are minimal or non-existent, causing training to stall. While less common as a catastrophic failure in Transformers compared to older RNNs (thanks to residual connections and normalization), it can still hinder convergence.
  • Learning Rate Appropriateness: The gradient norm interacts with the learning rate. A consistently high norm might suggest the learning rate needs reduction, while a consistently tiny norm might (in some cases) warrant an increase, although vanishing gradients are more often an architectural or initialization issue.

Expected Behavior: The gradient norm often starts relatively high and decreases as the model converges and the loss flattens near a minimum. However, its behavior is highly dependent on the learning rate schedule, optimizer, and data. Significant fluctuations can occur, but extremely large values are a red flag.

Signs of Instability:

  • Sudden Large Spikes: Values orders of magnitude larger than usual often precede NaN losses or significant loss spikes. This is the hallmark of exploding gradients.
  • Consistently Near Zero: If the gradient norm becomes extremely small early in training and stays there, it might indicate a vanishing gradient problem or issues with initialization.

Implementation: Calculating the total gradient norm requires iterating over all model parameters after the loss.backward() call but before the optimizer.step() call. Gradient clipping, a technique discussed later, often involves calculating this norm anyway.

import torch
import torch.nn as nn
from torch.nn.utils import clip_grad_norm_
# Assume model, data_loader, optimizer are defined, log_metric is defined

global_step = 0
# Define a max grad norm value for clipping (discussed in stabilization techniques)
max_grad_norm = 1.0

for batch in data_loader:
    optimizer.zero_grad()

    # --- Forward pass and loss calculation as before ---
    inputs = batch['input_ids'].to('cuda')
    targets = batch['labels'].to('cuda')
    outputs = model(inputs)
    loss_fct = nn.CrossEntropyLoss()
    loss = loss_fct(
        outputs.view(-1, model.config.vocab_size),
        targets.view(-1)
    )

    # Log loss
    if global_step % 10 == 0:
        log_metric(global_step, "train_loss", loss.item())

    if torch.isnan(loss):
        print(f"NaN loss detected at step {global_step}! Stopping training.")
        break

    loss.backward()

    # --- Calculate and log gradient norm BEFORE optimizer step ---
    # Create a generator of parameter gradients
    grads = [p.grad for p in model.parameters() if p.grad is not None]
    if len(grads) > 0:
      # Calculate the L2 norm over all gradients
      total_norm = torch.norm(
          torch.stack([
              torch.norm(g.detach(), 2.0) for g in grads
          ]),
          2.0
      )

      # Log the gradient norm (e.g., every 10 steps)
      if global_step % 10 == 0:
          log_metric(global_step, "grad_norm", total_norm.item())

      # Optional: Clip gradients (common practice)
      # clip_grad_norm_(model.parameters(), max_grad_norm)

      # Check for exploding gradients
      if total_norm > 100 * max_grad_norm: # Heuristic threshold
            print(
                f"Warning: High gradient norm ({total_norm:.2f}) "
                f"detected at step {global_step}"
            )
    else:
        # Handle case where there are no gradients
        # (e.g., if model has no trainable params)
        if global_step % 10 == 0:
            log_metric(global_step, "grad_norm", 0.0)


    optimizer.step()
    global_step += 1

Visualizing the gradient norm alongside the loss provides a more complete picture of training dynamics.

Example gradient L2 norm curves showing a stable decrease and an exploding gradient scenario where the norm increases dramatically.

By carefully monitoring both the training loss and the gradient norm, you gain essential visibility into the training process. These metrics act as early warning systems, allowing you to identify potential instabilities before they escalate into critical failures, saving valuable time and computational resources. Anomalies in these curves are the starting point for diagnosing the specific underlying issues, which we will explore next.

  • Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - Provides core theory on neural network training, including optimization, loss functions, and issues like exploding/vanishing gradients.
  • CS224n: Natural Language Processing with Deep Learning, Diyi Yang, Tatsunori Hashimoto, 2023 (Stanford University) - Course materials for deep learning in NLP, covering training practices, optimization, and monitoring for language models.
  • Scaling Laws for Neural Language Models, Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei, 2020 arXiv preprint arXiv:2001.08361 DOI: 10.48550/arXiv.2001.08361 - Research paper analyzing the behavior of training loss with model and dataset scale in large language models, setting expectations for performance.

© 2026 ApX Machine Learning