Motivation for Model Compression

Successfully training large language models frequently results in artifacts possessing billions, sometimes even trillions, of parameters. While these large models demonstrate remarkable capabilities, their sheer scale introduces significant practical hurdles when moving from the research lab or training cluster to applications. The necessity for model compression arises directly from these challenges, primarily centered around memory requirements, computational demands during inference, and the associated operational costs.

The Memory Bottleneck

Consider a model with 7 billion parameters. If each parameter is stored using standard 32-bit floating-point precision ($FP32$), which uses 4 bytes per parameter, the memory required just to load the model weights is substantial:

$$\text{Memory Usage} = \text{Number of Parameters} \times \text{Bytes per Parameter}$$ $$\text{Memory Usage} = 7 \times 10^9 \text{ parameters} \times 4 \text{ bytes/parameter} = 28 \times 10^9 \text{ bytes} = 28 \text{ GB}$$

This 28 GB calculation only accounts for the model weights themselves. During inference, additional memory is needed for activations, temporary computations, and the key-value (KV) cache (which we'll discuss in Chapter 28), especially when processing long sequences or handling multiple requests in batches. High-end GPUs often come with 24 GB, 40 GB, or 80 GB of High Bandwidth Memory (HBM), but even these can be insufficient for the largest models or for serving multiple model replicas efficiently. Requiring such high-memory hardware significantly increases deployment costs and limits the types of devices the model can run on.

import torch
import torch.nn as nn

# Example: Large linear layer dimensions
hidden_dim = 4096
intermediate_dim = 11008 # Common in ~7B models

# Parameters in just one Feed-Forward block's linear layers
ffn_layer1 = nn.Linear(hidden_dim, intermediate_dim, bias=False)
ffn_layer2 = nn.Linear(intermediate_dim, hidden_dim, bias=False)

params_ffn1 = hidden_dim * intermediate_dim
params_ffn2 = intermediate_dim * hidden_dim

total_ffn_params = params_ffn1 + params_ffn2
# A typical ~7B model has many such layers (~32), plus attention, embeddings...
print(f"Parameters in one FFN block (approx): {total_ffn_params:,}")
# Output: Parameters in one FFN block (approx): 90,177,536

# Estimate memory for these parameters in FP32
memory_ffn_gb = (total_ffn_params * 4) / (1024**3)
print(f"Memory for one FFN block (FP32, approx): {memory_ffn_gb:.2f} GB")
# Output: Memory for one FFN block (FP32, approx): 0.34 GB
# Multiply by number of layers (~32) -> ~10.8 GB just for FFN weights!

This simple calculation highlights how quickly memory requirements escalate, even before considering attention mechanisms and embedding tables.

Estimated memory required just to store model weights using 32-bit precision for different model sizes.

Inference Latency and Throughput

Beyond memory, the computational cost of running a forward pass through these massive networks is significant. Autoregressive generation, the standard way LLMs produce text, involves running the model sequentially for each token generated. While techniques like KV caching (Chapter 28) help, the sheer number of matrix multiplications and other operations required per token imposes a lower bound on latency.

For interactive applications like chatbots, coding assistants, or real-time translation, high latency leads to a poor user experience. Even for offline tasks like document summarization, slow inference speed increases the time required to process large datasets. Furthermore, high per-request latency limits the overall throughput (requests served per second) of a deployment, requiring more parallel hardware instances to handle a given load, again driving up costs.

Operational Costs and Accessibility

The combination of high memory requirements and significant computational demands translates directly into higher operational expenses:

1. Hardware Costs: Deploying LLMs often necessitates expensive, high-end GPUs or specialized accelerators. Scaling a service requires provisioning numerous such devices.
2. Energy Consumption: These powerful hardware components consume considerable amounts of electricity, contributing to both operational costs and environmental concerns.
3. Infrastructure Complexity: Managing clusters of specialized hardware for inference adds complexity to deployment and maintenance pipelines.

These factors create a barrier, making it difficult or impossible to deploy state-of-the-art LLMs in certain scenarios:

• On-Device Deployment: Running models directly on smartphones, laptops, or embedded systems is often infeasible due to strict memory, power, and thermal constraints. Compression is essential for enabling edge AI applications involving LLMs.
• Resource-Limited Settings: Researchers, startups, or organizations without access to large-scale cloud computing infrastructure may find deploying large models prohibitively expensive.
• Scalability: Even large organizations face challenges in scaling inference infrastructure cost-effectively to serve millions of users. More efficient models alleviate this burden.

Model compression techniques provide a pathway to mitigate these challenges. By reducing the memory footprint and computational requirements, we can:

• Deploy models on less expensive, more readily available hardware.
• Reduce inference latency, improving user experience in interactive applications.
• Lower operational costs (hardware, energy).
• Enable deployment in resource-constrained environments, including edge devices.
• Facilitate broader access to powerful language modeling capabilities.

The following sections will examine the primary methods for achieving these goals: quantization, pruning, and knowledge distillation. Each technique involves trade-offs, typically exchanging some degree of model performance for significant gains in efficiency. Understanding these methods and their implications is important for any engineer tasked with bringing large language models into production.