ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Practical Quantization for Large Language Models
Chapter 1: Foundations of Model Quantization
Introduction to Model Compression
Why Quantize Large Language Models?
Representing Numbers: Floating-Point vs. Fixed-Point
Integer Data Types in Quantization
Quantization Schemes: Symmetric vs. Asymmetric
Quantization Granularity Options
Measuring Quantization Error
Overview of Quantization Techniques
Quiz for Chapter 1
Chapter 2: Post-Training Quantization (PTQ)
Principles of Post-Training Quantization
Calibration: Selecting Representative Data
Static vs. Dynamic Quantization
Common PTQ Algorithms
Handling Outliers in PTQ
Applying PTQ to LLM Layers
Limitations of Basic PTQ
Hands-on Practical: Applying Static PTQ
Quiz for Chapter 2
Chapter 3: Advanced PTQ Techniques
Introduction to GPTQ
Understanding GPTQ Algorithm Mechanics
AWQ: Activation-aware Weight Quantization
SmoothQuant: Mitigating Activation Outliers
Comparing Advanced PTQ Methods
Implementation Considerations for Advanced PTQ
Hands-on Practical: Quantizing with GPTQ
Quiz for Chapter 3
Chapter 4: Quantization-Aware Training (QAT)
Need for Quantization-Aware Training
Simulating Quantization Effects During Training
Straight-Through Estimator (STE)
Implementing QAT with Deep Learning Frameworks
Fine-tuning Models with Quantization Nodes
Benefits and Drawbacks of QAT vs. PTQ
Practical Considerations for QAT Execution
Hands-on Practical: Setting up a Simple QAT Run
Quiz for Chapter 4
Chapter 5: Quantization Formats and Tooling
Overview of Common Quantized Model Formats
GGUF: Structure and Usage
GPTQ Format: Library Support and Implementation
AWQ Format Details
Working with Hugging Face Transformers and Optimum
Using bitsandbytes for Quantization
Tools for Model Conversion and Loading
Practice: Converting and Loading Quantized Formats
Quiz for Chapter 5
Chapter 6: Evaluating and Deploying Quantized LLMs
Metrics for Evaluating Quantized Models
Benchmarking Inference Speed and Memory Usage
Hardware Considerations for Quantized Inference
Deployment Strategies for Quantized LLMs
Troubleshooting Common Quantization Issues
Analyzing Accuracy vs. Performance Trade-offs
Practice: Benchmarking a Quantized LLM
Quiz for Chapter 6
Optimizing Large Language Models for Domain Specific Edge Deployment

Straight-Through Estimator (STE)

Quantization-Aware Training (QAT) simulates the effects of quantization during the training or fine-tuning process. This simulation happens by inserting operations, often called "fake quantization" nodes, into the model's computational graph. These nodes take high-precision inputs (like FP32 activations or weights) and produce low-precision outputs (simulating INT8 or INT4, for example), which are then used in the subsequent operations.

However, this introduces a significant challenge for training algorithms that rely on gradient descent. The core operation within fake quantization is the rounding function, which maps a continuous range of inputs to a discrete set of output values. Mathematically, this function is a step function.

For example, a simple rounding function, Round(x), has a derivative that is zero almost everywhere (on the flat steps) and undefined at the points where the value jumps. Standard backpropagation relies on calculating gradients to update model weights. If the gradient is zero or undefined, the updates cannot flow back through the quantization operation to the preceding layers and weights. This effectively stalls the learning process for parameters situated before the quantization node.

How can the model learn to adapt to quantization if the gradients are blocked? This is where the Straight-Through Estimator (STE) comes into play.

The Problem: Non-Differentiable Quantization

Let's represent the quantization function (including scaling, rounding, and de-quantization back to a float that mimics the low-precision step) as y=q(x)y = q(x)y=q(x). In the forward pass of training, we compute yyy from the input xxx and use yyy in subsequent calculations.

During the backward pass, we need to compute the gradient of the loss LLL with respect to the input xxx, denoted as ∂L∂x\frac{\partial L}{\partial x}∂x∂L​. Using the chain rule, this would normally be calculated as:

∂L∂x=∂L∂y⋅∂y∂x\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial x}∂x∂L​=∂y∂L​⋅∂x∂y​

The problem lies in the term ∂y∂x=∂q(x)∂x\frac{\partial y}{\partial x} = \frac{\partial q(x)}{\partial x}∂x∂y​=∂x∂q(x)​. As mentioned, this derivative is problematic (mostly 0, undefined at jumps). If we use this true gradient, ∂L∂x\frac{\partial L}{\partial x}∂x∂L​ becomes zero almost everywhere, preventing weight updates.

The Solution: Straight-Through Estimator (STE)

The Straight-Through Estimator (STE) provides a practical workaround for this issue. It's an approximation used specifically during the backward pass. The core idea is simple:

  1. Forward Pass: Compute the quantization operation as usual: y=q(x)y = q(x)y=q(x). The effects of quantization (precision loss) are fully applied.
  2. Backward Pass: When computing the gradient ∂L∂x\frac{\partial L}{\partial x}∂x∂L​, ignore the true derivative of q(x)q(x)q(x) and instead approximate it. The most common STE approximation sets the gradient ∂y∂x\frac{\partial y}{\partial x}∂x∂y​ to 1.

Therefore, during backpropagation with STE, the gradient calculation becomes:

∂L∂x≈∂L∂y⋅1=∂L∂y\frac{\partial L}{\partial x} \approx \frac{\partial L}{\partial y} \cdot 1 = \frac{\partial L}{\partial y}∂x∂L​≈∂y∂L​⋅1=∂y∂L​

This means the gradient ∂L∂y\frac{\partial L}{\partial y}∂y∂L​ computed for the output of the quantization node is passed "straight through" to become the gradient ∂L∂x\frac{\partial L}{\partial x}∂x∂L​ for the input, as if the quantization operation was an identity function (y=xy=xy=x) only for the gradient calculation.

The diagram illustrates the STE process. The forward pass applies the actual quantization q(x)q(x)q(x). The backward pass uses the STE approximation (∂y∂x≈1\frac{\partial y}{\partial x} \approx 1∂x∂y​≈1) to allow the incoming gradient ∂L∂y\frac{\partial L}{\partial y}∂y∂L​ to pass through unchanged to become ∂L∂x\frac{\partial L}{\partial x}∂x∂L​.

Why Does STE Work?

It might seem counter-intuitive to use an approximation that ignores the true nature of the quantization function during backpropagation. However, STE works well in practice for several reasons:

  • Gradient Flow: It ensures that gradients are not blocked, allowing optimization signals to reach the weights preceding the quantization node.
  • Learning Adaptation: Although the gradient calculation is an approximation, the forward pass still uses the actual quantized values. The loss function reflects the errors introduced by this quantization. By allowing gradients to flow, STE enables the model weights to adjust in a way that minimizes this quantization-induced error. The network learns parameters that are more resilient to the rounding effects applied during the forward pass.
  • Simplicity: STE is easy to implement within standard deep learning frameworks. Most frameworks providing QAT capabilities incorporate STE internally.

Variants of STE

While the identity approximation (∂y∂x=1\frac{\partial y}{\partial x} = 1∂x∂y​=1) is the most common form of STE, some variations exist. For instance, another approach involves clipping the gradient based on the input range used for quantization. If the quantization function effectively clips inputs xxx to a range [cmin,cmax][c_{min}, c_{max}][cmin​,cmax​] before quantizing, the STE might be defined as:

∂y∂x={1if cmin≤x≤cmax0otherwise\frac{\partial y}{\partial x} = \begin{cases} 1 & \text{if } c_{min} \le x \le c_{max} \\ 0 & \text{otherwise} \end{cases}∂x∂y​={10​if cmin​≤x≤cmax​otherwise​

This variant prevents gradients from flowing back for inputs that were already outside the quantization range, which can sometimes help stabilize training. However, the simple identity approximation often suffices.

In summary, the Straight-Through Estimator is a fundamental technique that makes Quantization-Aware Training possible. By providing a path for gradients through the non-differentiable quantization operations, it allows deep learning models to adapt their weights during training or fine-tuning, leading to significantly better accuracy for quantized models compared to Post-Training Quantization, especially at very low bit-widths.

  • Quantized Neural Networks: Training Neural Networks with Low-Precision Weights and Activations, Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio, 2016 Advances in Neural Information Processing Systems, Vol. 29 (Curran Associates, Inc.) - This influential paper applies the Straight-Through Estimator to train neural networks with extremely low-precision weights and activations, laying important groundwork for Quantization-Aware Training.
  • Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko, 2018 2018 IEEE Conference on Computer Vision and Pattern Recognition (IEEE) DOI: 10.1109/CVPR.2018.00794 - A seminal paper describing the Quantization-Aware Training (QAT) method, which explicitly uses Straight-Through Estimators to simulate quantization during training, enabling high-accuracy low-bit inference.

© 2026 ApX Machine Learning