APX AI
Online
Inference execution splits into two distinct operational phases with different hardware bottlenecks: the compute-bound prefill phase that processes prompt tokens in parallel, and the memory-bandwidth-bound decode phase that generates output tokens autoregressively. Managing dynamic runtime memory during these phases requires tracking activation buffers and the Key-Value (KV) cache as context length and batch concurrency scale.
This chapter details the mechanics of attention variants including Multi-Head Attention, Grouped-Query Attention, and Multi-Head Latent Attention, along with runtime optimization techniques such as PagedAttention, continuous batching, and speculative decoding. These concepts establish how memory allocation and compute utilization behave under real-time serving workloads.
3.1 Prefill Phase Compute Operations
3.2 Decode Phase Memory Bandwidth Constraints
3.3 Attention Topologies MHA GQA MQA and MLA
3.4 Sliding Window and Linear Attention Models
3.5 KV Cache Memory Scaling Mechanics
3.6 PagedAttention Memory Management
3.7 Continuous Batching Implementation
3.8 Speculative Decoding Systems