ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

Masterclass

How To Build A Large Language Model
Chapter 1: Introduction to Large-Scale Language Modeling
Defining Large Language Models
Historical Context of Sequence Modeling
The Significance of Scale
Computational Challenges Overview
Software and Hardware Ecosystem
Chapter 2: Mathematical Preliminaries for LLMs
Linear Algebra Review: Vectors and Matrices
Calculus Review: Gradients and Optimization
Probability and Statistics Fundamentals
Numerical Stability Considerations
Notation Used Throughout This Course
Chapter 3: Revisiting Sequence Processing Architectures
Fundamentals of Recurrent Neural Networks (RNNs)
Limitations of Simple RNNs
Long Short-Term Memory (LSTM) Networks
Gated Recurrent Units (GRUs)
Sequence-to-Sequence Models with RNNs
Chapter 4: The Transformer Architecture
Overcoming Recurrence with Attention
Scaled Dot-Product Attention
Multi-Head Attention Mechanism
Positional Encoding Techniques
Encoder and Decoder Stacks
The Role of Layer Normalization and Residual Connections
Chapter 5: Tokenization for Large Vocabularies
The Need for Subword Tokenization
Byte Pair Encoding (BPE) Algorithm
WordPiece Tokenization
SentencePiece Implementation
Handling Special Tokens
Vocabulary Size Selection Trade-offs
Chapter 6: Sourcing and Acquiring Massive Text Datasets
Identifying Potential Data Sources
Utilizing Common Crawl Data
Web Scraping Techniques at Scale
Leveraging Open Licensed Datasets
Data Acquisition Legal Considerations
Chapter 7: Data Cleaning and Preprocessing Pipelines
Strategies for Quality Filtering
Text Normalization Methods
Handling Boilerplate and Markup Removal
Near-Duplicate and Exact Duplicate Detection
Language Identification and Filtering
Building Scalable Preprocessing Pipelines
Chapter 8: Building and Managing Large-Scale Datasets
Data Storage Formats (Text, Arrow, Parquet)
Distributed File Systems (HDFS, S3)
Data Indexing for Efficient Retrieval
Dataset Versioning and Reproducibility
Streaming Data Loaders for Training
Chapter 9: Data Sampling Strategies for Training
Importance of Data Mixture Composition
Source Weighting Strategies
Temperature-Based Sampling
Introduction to Curriculum Learning
Data Pacing and Annealing Schedules
Chapter 10: Implementing the Transformer from Scratch
Setting up the Project Environment
Implementing Scaled Dot-Product Attention
Building the Multi-Head Attention Layer
Implementing the Position-wise Feed-Forward Network
Constructing the Encoder and Decoder Layers
Assembling the Full Transformer Model
Chapter 11: Scaling Transformers: Architectural Choices
Scaling Laws for Neural Language Models
Depth vs Width Trade-offs
Choice of Activation Functions (ReLU, GeLU, SwiGLU)
Normalization Layer Placement (Pre-LN vs Post-LN)
Introduction to Sparse Attention Mechanisms
Chapter 12: Initialization Techniques for Deep Networks
The Importance of Proper Initialization
Xavier (Glorot) Initialization
Kaiming (He) Initialization
Initialization in Transformer Components
Small Initialization for Final Layers
Chapter 13: Positional Encoding Variations
Limitations of Absolute Positional Encodings
Relative Positional Encoding Concepts
Implementation of Shaw et al.'s Relative Position
Transformer-XL Relative Positional Encoding
Rotary Position Embedding (RoPE)
Chapter 14: Advanced Architectural Modifications
Parameter-Efficient Fine-Tuning Needs
Adapter Modules for Transformers
Introduction to Mixture-of-Experts (MoE)
Routing Mechanisms in MoE
Load Balancing in MoE Layers
Chapter 15: Distributed Training Strategies
Motivation: Why Distributed Training?
Data Parallelism (DP)
Tensor Parallelism (TP)
Pipeline Parallelism (PP)
Hybrid Approaches (DP+TP, DP+PP, etc.)
Communication Overhead Analysis
Chapter 16: Implementing Distributed Training Frameworks
Overview of Distributed Training Libraries
Introduction to DeepSpeed
Using DeepSpeed ZeRO Optimizations
Introduction to Megatron-LM
Configuring Tensor and Pipeline Parallelism in Megatron-LM
Combining Frameworks and Strategies
Chapter 17: Optimization Algorithms for LLMs
Review of Gradient Descent Variants (SGD, Momentum)
Adaptive Optimizers: Adam and AdamW
Learning Rate Scheduling Strategies
Gradient Clipping Techniques
Choosing Optimizer Hyperparameters (lr, betas, eps, weight_decay)
Chapter 18: Hardware Considerations for LLM Training
GPU Architectures (NVIDIA Ampere, Hopper)
TPU Architectures (Google TPUs)
Memory Requirements (HBM, GPU RAM)
Interconnect Technologies (NVLink, InfiniBand)
Hardware Selection Trade-offs (Cost, Performance, Availability)
Chapter 19: Checkpointing and Fault Tolerance
The Need for Checkpointing in Long Training Runs
Saving Model State (Weights, Optimizer States)
Handling Distributed Checkpointing
Asynchronous vs Synchronous Checkpointing
Checkpoint Frequency and Storage Management
Resuming Training from Checkpoints
Chapter 20: Mixed-Precision Training Techniques
Introduction to Floating-Point Formats (FP32, FP16, BF16)
Benefits of Lower Precision (Speed, Memory)
Challenges with FP16 Training (Range Issues)
Loss Scaling Techniques
Using BF16 (BFloat16) Format
Framework Support for Mixed Precision (AMP)
Chapter 21: Intrinsic Evaluation Metrics
Concept of Language Model Evaluation
Perplexity: Definition and Calculation
Interpreting Perplexity Scores
Bits Per Character/Word
Effect of Tokenization on Perplexity
Chapter 22: Extrinsic Evaluation on Downstream Tasks
Rationale for Downstream Task Evaluation
Common Downstream NLP Tasks
Fine-tuning Procedures for Evaluation
Standard Benchmarks: GLUE and SuperGLUE
Few-Shot and Zero-Shot Evaluation
Developing Custom Evaluation Tasks
Chapter 23: Analyzing Model Behavior
Challenges in Interpreting LLMs
Attention Map Visualization
Probing Internal Representations
Neuron Activation Analysis
Identifying Failure Modes
Chapter 24: Identifying and Mitigating Training Instabilities
Common Symptoms of Instability
Monitoring Training Metrics (Loss, Grad Norm)
Diagnosing Loss Spikes
Debugging Numerical Precision Issues
Stabilization Techniques Revisited (Clipping, LR, Warmup)
Impact of Architectural Choices on Stability
Chapter 25: Fine-tuning for Alignment: Supervised Fine-Tuning (SFT)
Goals of LLM Alignment
Introduction to Supervised Fine-Tuning (SFT)
Creating High-Quality Instruction Datasets
Data Formatting for SFT (Prompts, Completions)
The SFT Training Process and Hyperparameters
Evaluating SFT Models on Alignment Goals
Chapter 26: Reinforcement Learning from Human Feedback (RLHF)
The RLHF Pipeline Overview
Collecting Human Preference Data
Training the Reward Model (RM)
Introduction to Proximal Policy Optimization (PPO)
RL Fine-tuning with PPO
The Role of the KL Divergence Penalty
Challenges and Considerations in RLHF
Alternatives: Direct Preference Optimization (DPO)
Chapter 27: Model Compression Techniques
Motivation for Model Compression
Weight Quantization (INT8, INT4)
Activation Quantization Considerations
Network Pruning (Structured vs Unstructured)
Knowledge Distillation
Evaluating Performance vs Compression Trade-offs
Chapter 28: Efficient Inference Strategies
Challenges in Autoregressive Decoding
Key-Value (KV) Caching
Optimized Attention Implementations (FlashAttention)
Batching Strategies for Throughput
Speculative Decoding
Chapter 29: Serving LLMs at Scale
API Design for LLM Interaction
Model Serving Frameworks (Triton, TorchServe)
Handling Concurrent Requests
Load Balancing Across Model Instances
Monitoring Serving Performance and Cost
Chapter 30: Continuous Training and Model Updates
Motivation for Continuous Improvement
Strategies for Continual Pre-training
Strategies for Continual Fine-Tuning (SFT/RLHF)
Incorporating New Data Sources Safely
Updating Models with Architectural Changes
Versioning, Deployment, and Rollback Strategies

Effect of Tokenization on Perplexity

Perplexity serves as a fundamental intrinsic metric for evaluating language models, quantifying how well the model predicts a given sequence of text. It's derived directly from the model's assigned probabilities to the tokens in that sequence. A significant point often overlooked is that the calculated perplexity is highly sensitive to the specific tokenization scheme used. The perplexity formula is:

PPL(W)=exp⁡(−1N∑i=1Nlog⁡p(wi∣w<i;θ))PPL(W) = \exp\left(-\frac{1}{N}\sum_{i=1}^N \log p(w_i | w_{<i}; \theta)\right)PPL(W)=exp(−N1​∑i=1N​logp(wi​∣w<i​;θ))

Here, wiw_iwi​ represents the iii-th token in the sequence, and NNN is the total number of tokens. This means that both the individual probabilities p(wi∣w<i;θ)p(w_i | w_{<i}; \theta)p(wi​∣w<i​;θ) and the sequence length NNN are direct functions of how the raw text is segmented into tokens.

For example, a simple sentence: "Tokenization impacts perplexity."

Let's see how different tokenizers might handle this:

  1. Word-Level Tokenizer: Might produce ["Tokenization", "impacts", "perplexity", "."] -> N=4N = 4N=4 tokens. The model predicts the probability of "impacts" given "Tokenization", then "perplexity" given the first two, and so on.
  2. Character-Level Tokenizer: Would produce ['T', 'o', 'k', 'e', 'n', 'i', 'z', 'a', 't', 'i', 'o', 'n', ' ', 'i', 'm', 'p', 'a', 'c', 't', 's', ' ', 'p', 'e', 'r', 'p', 'l', 'e', 'x', 'i', 't', 'y', '.'] -> N=33N = 33N=33 tokens. The model predicts 'o' given 'T', 'k' given 'To', etc. The prediction task is very different.
  3. Subword Tokenizer (e.g., BPE/WordPiece): Might produce ["Token", "ization", "Ġimpacts", "Ġperplex", "ity", "."] -> N=6N = 6N=6 tokens (assuming a GPT-2 style tokenizer where Ġ indicates a space). Here, the model predicts "ization" given "Token", "Ġimpacts" given "Tokenization", and so forth.

This example highlights two main effects:

  • Sequence Length (N): Different tokenizers yield different numbers of tokens (NNN) for the same underlying text. Since perplexity involves averaging the negative log probability over NNN, changing NNN directly alters the final score, even if the underlying total probability assigned to the full sequence remained constant (which it usually doesn't). Generally, character-level tokenization results in much longer sequences (higher NNN) compared to subword or word tokenization for the same text.
  • Prediction Task Difficulty: The nature of the prediction task changes. Predicting the next character is typically easier (lower conditional entropy) than predicting the next whole word or complex subword. However, a character model needs to make many more predictions. A word-level model makes fewer predictions, but each prediction might be harder (drawing from a larger vocabulary, predicting longer units). Subword models sit somewhere in between.

Let's illustrate the token difference with a brief PyTorch example using the transformers library:

import torch
from transformers import AutoTokenizer

# Load two different tokenizers
tokenizer_bert = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenizer_gpt2 = AutoTokenizer.from_pretrained('gpt2')

text = "Tokenization impacts perplexity."

# Tokenize with BERT tokenizer (WordPiece)
tokens_bert = tokenizer_bert.tokenize(text)
ids_bert = tokenizer_bert.encode(text)
print(f"BERT Tokens ({len(tokens_bert)}): {tokens_bert}")
# Output: BERT Tokens (6): ['token', '##ization', 'impacts', 'per',
# '##plex', '##ity', '.']
print(f"BERT IDs ({len(ids_bert)}): {ids_bert}")
# Output: BERT IDs (9): [101, 19204, 17260, 7296, 2361, 18049, 4234,
# 1012, 102]
# Note: Includes [CLS] and [SEP] tokens

# Tokenize with GPT-2 tokenizer (BPE)
tokens_gpt2 = tokenizer_gpt2.tokenize(text)
ids_gpt2 = tokenizer_gpt2.encode(text)
print(f"GPT-2 Tokens ({len(tokens_gpt2)}): {tokens_gpt2}")
# Output: GPT-2 Tokens (6): ['Token', 'ization', 'Ġimpacts', 'Ġperplex',
# 'ity', '.']
print(f"GPT-2 IDs ({len(ids_gpt2)}): {ids_gpt2}")
# Output: GPT-2 IDs (6): [11934, 10004, 33333, 21119, 2138, 13]
# Note: GPT-2 tokenizer doesn't add special tokens by default here,
# length matches tokens

Notice that even between two subword tokenizers (BERT's WordPiece and GPT-2's BPE), the segmentation differs ('token', '##ization' vs 'Token', 'ization'), and the inclusion of special tokens like [CLS] and [SEP] by default in BERT's encode method affects the sequence length (NNN) used in a standard perplexity calculation.

The direct consequence is that perplexity scores are only directly comparable between models if they use the exact same tokenizer and vocabulary on the evaluation dataset. Comparing the perplexity of a model using BPE with 50,000 merges to one using WordPiece with a 30,000-word vocabulary is like comparing apples and oranges. The underlying units of prediction are different.

The chart above illustrates how perplexity scores for the same underlying text might vary dramatically just based on the chosen tokenizer. Lower values suggest the model finds the prediction task easier per token, but this doesn't allow direct comparison of model quality across different tokenizations.

Furthermore, preprocessing steps applied before tokenization, such as lowercasing or Unicode normalization, also interact with this. If one evaluation lowercases the text and another does not, a case-sensitive tokenizer will produce different tokens, leading to incomparable perplexity scores.

When reporting or interpreting perplexity, always ensure you know exactly which tokenizer was used, including its vocabulary size and any associated preprocessing steps. Without this context, a raw perplexity number offers limited insight into the model's relative capabilities compared to others evaluated under different tokenization regimes. The most reliable comparisons are made when evaluating different models or checkpoints using an identical evaluation setup, including the tokenizer.

  • Neural Machine Translation of Rare Words with Subword Units, Rico Sennrich, Barry Haddow, and Alexandra Birch, 2016 Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Association for Computational Linguistics) DOI: 10.18653/v1/P16-1162 - Introduces Byte-Pair Encoding (BPE) for subword tokenization, a foundational method used in many modern language models like GPT-2, directly relevant to the subword examples discussed.
  • SentencePiece: A language independent subword tokenizer and detokenizer for Neural Text Processing, Taku Kudo and John Richardson, 2018 Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (Association for Computational Linguistics) DOI: 10.18653/v1/D18-2012 - Describes SentencePiece, a toolkit that implements subword tokenization algorithms including WordPiece, important for understanding how tokenizers like BERT's operate.
  • Tokenizers in the transformers library, Hugging Face team, 2024 - Official documentation for tokenizers within the Hugging Face transformers library, explaining how different tokenization algorithms are loaded and used, directly relevant to the Python code example.

© 2026 ApX Machine Learning