Optimizing a large language model through techniques like quantization or pruning addresses model size and computational complexity. However, achieving significant inference speedups requires effective interaction between the model and the underlying hardware and system software. This chapter focuses on the methods used to accelerate LLM execution by tailoring computations to specific hardware capabilities and optimizing the surrounding system components.

You will learn about:

• Mapping core LLM operations, such as attention and matrix multiplication, onto diverse hardware architectures including CPUs, GPUs, and specialized accelerators.
• Strategies for managing the substantial memory requirements of LLMs, including techniques like activation checkpointing and efficient caching.
• Utilizing specialized, highly optimized software routines (kernels) for core LLM computations to maximize hardware throughput.
• How compilers contribute to performance through techniques like operator fusion and memory layout optimization.
• Approaches for distributing inference across multiple devices using tensor and pipeline parallelism.
• Algorithms designed specifically to speed up the generation process, such as speculative decoding.
• Systematic methods for benchmarking performance across different hardware platforms.
• Practical application of optimization frameworks and runtimes like ONNX Runtime, TensorRT, or vLLM.

Understanding these hardware and systems-level optimizations helps bridge the gap between a theoretically compressed model and a practically fast deployment.