PagedAttention Memory Management

Generating text with large language models requires caching previous token states to prevent redundant matrix multiplications during the decoding phase. When you load a model onto a physical device like a 24GB RTX 4090 or an 80GB A100, the static parameter weights consume a fixed footprint. The remaining memory is dedicated to the dynamic Key-Value (KV) cache and temporary framework buffers. Historically, inference engines allocated this cache contiguously based on the maximum possible sequence length configured for a request. If an application accepted prompts up to 4,096 tokens, the engine reserved exactly 4,096 tokens worth of continuous memory the moment the request started. If the user only sent a 100-token prompt and generated 50 tokens, the system left 3,946 tokens worth of allocated memory completely empty. This contiguous allocation method creates severe memory fragmentation and artificially restricts the number of concurrent users you can serve. PagedAttention solves this bottleneck by applying the principles of operating system virtual memory paging to GPU memory tensors.

Internal and External Fragmentation Constraints

Contiguous memory allocation introduces two distinct forms of waste that degrade hardware utilization. Internal fragmentation occurs because the exact output length of an autoregressive generation is unknown ahead of time. System operators must guess and pre-allocate the maximum expected generation length. The unused reserved space becomes dead memory.

External fragmentation occurs as different requests complete at different times. If you have a sequence of short, long, and medium requests processing concurrently, their completion leaves physical memory gaps of varying sizes. A new request requiring a large contiguous memory block cannot fit into these scattered gaps, even if the total free memory across the GPU is technically sufficient. In standard implementations without paging, these fragmentation issues routinely strand 60% to 80% of available VRAM, acting as an absolute limit on continuous batching performance.

PagedAttention breaks the KV cache down into non-contiguous physical blocks, typically holding 16 or 32 tokens each. The inference engine maintains a block table that maps logical token positions to these scattered physical blocks. The GPU only allocates physical memory as generation progresses and blocks fill up, eliminating internal waste and making external fragmentation irrelevant.

Calculating Physical Block Sizes

To size a cluster effectively, you must calculate exactly how much memory a single PagedAttention block consumes. The size of a block depends strictly on the model architecture and the precision format loaded into memory.

The formula for a single physical block in bytes is: $Block\_Size_{bytes} = 2 \times L \times N_{kv} \times D_{head} \times T_{block} \times P_{bytes}$

Where:

• $L$ is the number of transformer layers.
• $N_{kv}$ is the number of KV attention heads.
• $D_{head}$ is the dimension of each attention head.
• $T_{block}$ is the number of tokens stored per block.
• $P_{bytes}$ is the byte size of the data type (2 for FP16/BF16, 1 for FP8).
• The leading $2$ accounts for the distinct Key and Value tensors.

For example, deploying Qwen/Qwen2.5-7B in FP16 on a single RTX 3090 24GB GPU. This architecture features 28 layers, 4 KV heads, and a head dimension of 128. If we configure a block size of 16 tokens:

$Block\_Size_{bytes} = 2 \times 28 \times 4 \times 128 \times 16 \times 2 = 917,504 \text{ bytes (0.917 MB)}$

A 7.6 billion parameter model in FP16 requires approximately 15.2 GB for the static weights. Reserving 1.0 GB for the CUDA context leaves 7.8 GB of VRAM entirely for the KV cache blocks.

Dividing 7.8 GB by the 0.917 MB block size reveals that this GPU can hold roughly 8,500 physical blocks. Multiplying by 16 tokens per block yields a maximum concurrent token capacity of 136,000 tokens. You can distribute these 136,000 tokens across few users with massive context lengths, or hundreds of users with short context lengths, with less than 4% memory waste.

Mapping logical sequence tokens to non-contiguous physical VRAM blocks using a block table.

Production Implementation with vLLM

The most direct way to leverage PagedAttention in production is through inference engines like vLLM. Because the paging system manages memory securely, you can instruct the engine to allocate nearly all available GPU memory upfront. This prevents the PyTorch memory allocator from fighting the OS during runtime.

Configuring the engine with explicit block sizes and allocation pools establishes strict PagedAttention boundaries:

from vllm import LLM, SamplingParams

# Configure the LLM engine with PagedAttention constraints
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-8B",
    tensor_parallel_size=1,
    # Pre-allocate 90% of VRAM to the block pool to prevent OOM
    gpu_memory_utilization=0.90, 
    # Define exact tokens per block (affects mapping overhead vs waste)
    block_size=16,               
    # Set max concurrent requests the block table will track
    max_num_seqs=256,
    # Enforce FP16 precision
    dtype="float16"
)

prompts = [
    "Explain the architecture of a GPU memory hierarchy.",
    "Write a Python script for matrix multiplication."
]

sampling_params = SamplingParams(temperature=0.7, max_tokens=150)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt}\nOutput: {output.outputs[0].text}\n")

Setting gpu_memory_utilization to 0.90 is standard practice when using PagedAttention. Because external fragmentation is handled by the block table, the engine guarantees it will not exceed this limit, preventing unexpected Out Of Memory crashes during sudden traffic spikes.

Complex Topologies and Memory Sharing

Solving fragmentation, tracking physical memory in blocks enables zero-copy memory sharing for complex inference topologies. When a user requests multiple generated variations from a single prompt (parallel sampling) or when the engine executes beam search, the initial prefill prompt is identical across all parallel sequences.

In a contiguous allocation system, the engine must duplicate the prompt's KV cache for every single output sequence. If a user sends a 2,000-token prompt and asks for 5 distinct answers, the engine duplicates the 2,000-token KV cache 5 times, burning through memory rapidly.

PagedAttention handles this natively using reference counting. The block table maps all 5 logical sequences to the exact same physical blocks in VRAM for the prompt portion. The physical blocks maintain a reference count of 5. As the model begins the decode phase and generates different tokens for each sequence, it applies a Copy-On-Write mechanism. It only allocates new, distinct physical blocks for the newly generated tokens. This specific mechanic reduces memory overhead by 40% to 60% in multi-agent routing systems and structured JSON generation tasks where large context prefixes are shared across thousands of concurrent calls.