Home Blog Learn

All Courses

Deploying Quantized LLMs for Efficient Inference

Chapter 1: Advanced LLM Quantization Fundamentals

Revisiting Quantization Principles for Large Models

Low-Bit Quantization Techniques (Below INT8)

Understanding Quantization Data Types and Formats

Post-Training Quantization (PTQ) Algorithms for LLMs

Quantization-Aware Training (QAT) Considerations

Mixed-Precision Quantization Strategies

Calibration Data Selection and Preparation

Hands-on Practical: Applying GPTQ to an LLM

Chapter 2: Implementing LLM Quantization with Toolkits

Overview of LLM Quantization Libraries

Using bitsandbytes for Low-Bit Operations

Quantization with Hugging Face Transformers and Accelerate

Applying GPTQ using AutoGPTQ

Applying AWQ using AutoAWQ

Comparing Toolkit Outputs and Performance

Handling Model Compatibility Issues

Practice: Quantizing Models with Multiple Toolkits

Chapter 3: Performance Evaluation of Quantized LLMs

Metrics for Quantized Model Evaluation

Measuring Inference Latency and Throughput

Assessing Memory Consumption (Disk and Runtime)

Evaluating Accuracy Degradation

Benchmarking Frameworks and Tools

Analyzing Performance on Target Hardware

Visualizing Performance Trade-offs

Hands-on Practical: Benchmarking a Quantized LLM

Chapter 4: Optimizing and Deploying Quantized LLMs

Inference Optimization Techniques Post-Quantization

Choosing the Right Deployment Framework

Deploying with Text Generation Inference (TGI)

Leveraging vLLM for High-Throughput Inference

GPU Optimization with NVIDIA TensorRT-LLM

Deployment using ONNX Runtime

Containerization and Scaling Strategies

Monitoring Deployed Quantized Models

Hands-on Practical: Deploying via an Inference Server

Chapter 5: Addressing Advanced Challenges

Mitigating Accuracy Loss in Low-Bit Regimes

Handling Activation and Weight Outliers

Quantizing Specific LLM Components (Attention, Normalization)

Hardware Constraints and Kernel Availability

Dynamic Quantization vs. Static Quantization Trade-offs

Debugging Quantization Issues

Integrating Quantized Models into Production Pipelines

Practice: Fine-tuning Quantization Parameters

Performance Evaluation of Quantized LLMs

Sections

3.1 Metrics for Quantized Model Evaluation
3.2 Measuring Inference Latency and Throughput
3.3 Assessing Memory Consumption (Disk and Runtime)
3.4 Evaluating Accuracy Degradation
3.5 Benchmarking Frameworks and Tools
3.6 Analyzing Performance on Target Hardware
3.7 Visualizing Performance Trade-offs
3.8 Hands-on Practical: Benchmarking a Quantized LLM

© 2025 ApX Machine Learning

;