Continuous Batching Implementation

Legacy inference systems treat request batching exactly like training loops. They group a fixed number of prompts together, pad the shorter sequences to match the longest one, and process them in unison. This static approach results in massive inefficiencies during text generation. Because generation is an autoregressive process where each token depends on the previous ones, requests finish at entirely different times. In a statically batched system, the GPU wastes expensive compute cycles waiting for the longest sequence to complete while the completed requests sit idle holding onto allocated memory.

Continuous batching, often called iteration-level scheduling, solves this hardware underutilization by decoupling the lifecycle of individual requests. Instead of waiting for an entire batch to finish, the inference engine evaluates the active queue at every single token generation step (an iteration). When one request generates its final stop token, the system immediately evicts its data from GPU memory and injects a new request from the queue into the very next forward pass. This ensures the GPU Tensor Cores remain heavily saturated and memory bandwidth is amortized across as many active tokens as possible.

The Mechanics of Iteration-Level Scheduling

To design infrastructure that supports continuous batching, operators must understand how the engine mixes two completely different workload profiles in the same execution step.

1. The Prefill Phase: A compute-bound operation where the model processes a new prompt all at once. It relies on massive matrix-matrix multiplications (GEMM).
2. The Decode Phase: A memory-bandwidth-bound operation where the model generates one token at a time. It relies on matrix-vector multiplications (GEMV) and scales poorly unless batched.

Continuous batching systems must safely multiplex these operations. When a new request enters the batch, its large prompt (prefill) is processed alongside the single-token generation (decode) of the ongoing requests. If you allow too many prefill tokens into a single iteration, the compute time spikes, delaying the decode generation for the other active users and ruining your Time-Per-Output-Token (TPOT) SLAs.

Diagram mapping the token-by-token iteration lifecycle where completed sequences are immediately replaced by new requests without waiting for the entire batch to finish.

Memory Fragmentation and KV Cache Constraints

You cannot implement continuous batching safely without dynamic memory management, specifically PagedAttention. Because you do not know how many tokens a user will generate ahead of time, pre-allocating contiguous memory blocks for the maximum sequence length will instantly exhaust your VRAM. PagedAttention allocates KV cache in fixed-size non-contiguous blocks, allowing the continuous batching scheduler to request VRAM on demand as sequences grow token by token.

To determine how many concurrent sequences your hardware can sustain in a continuous batch, you must calculate the exact KV cache memory footprint required per iteration. The analytical equation for KV cache sizing is:

$Memory_{KV} = 2 \times num\_layers \times num\_kv\_heads \times head\_dim \times seq\_length \times bytes\_per\_element \times batch\_size$

For example, in a production setup running meta-llama/Meta-Llama-3.1-8B on a single RTX 4090 24GB VRAM GPU. The architecture specifications are 32 layers, 8 KV heads, and a head dimension of 128. If we load the model in FP16 precision (2 bytes per parameter), the weights consume roughly 16 GB of VRAM. This leaves approximately 8 GB for the KV cache and CUDA context overhead.

If we attempt to support a continuous batch size of 16 concurrent users, each reaching a 4096 sequence length, the calculation is:

$Memory_{KV} = 2 \times 32 \times 8 \times 128 \times 4096 \times 2 \times 16$

This equals 8,589,934,592 bytes, or exactly 8.58 GB. Since our RTX 4090 only has 8 GB of free VRAM remaining, the inference engine will crash with an Out Of Memory (OOM) error before the batch finishes. To solve this physical hardware constraint, operators have three choices. First, drop the maximum concurrent batch size to 8. Second, offload the model to a dual-GPU 48GB setup (like two RTX 3090s) using Tensor Parallelism. Third, apply 8-bit quantization to the KV cache, which halves the bytes_per_element to 1 byte, reducing the KV cache footprint to 4.29 GB and safely fitting within the 24GB limit.

Implementing Continuous Batching in Code

Modern serving engines like vLLM abstract the complexity of PagedAttention and iteration-level scheduling. When configuring the engine, you define strict upper boundaries so the scheduler knows exactly how to pack the continuous batch without exceeding VRAM or latency SLAs.

from vllm import LLM, SamplingParams

# Initialize the continuous batching engine
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-8B",
    tensor_parallel_size=1,
    # The maximum total tokens (prefill + decode) processed in a single iteration
    max_num_batched_tokens=4096,
    # The maximum number of concurrent requests in the batch
    max_num_seqs=8,
    # Allocate 90% of free VRAM to the PagedAttention KV cache pool
    gpu_memory_utilization=0.90,
    dtype="float16"
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
prompts = [
    "Write a Python script for file parsing.",
    "Explain the mechanics of CPU caching.",
    "Translate this text to French."
]

# The engine processes these asynchronously using iteration-level scheduling
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt[:20]}... Generated: {len(output.outputs[0].token_ids)} tokens")

The parameters max_num_batched_tokens and max_num_seqs govern the scheduler's behavior. If a new request arrives with a 5000-token prompt, but max_num_batched_tokens is 4096, the scheduler will chunk the prefill phase across multiple iterations. This prevents massive prompts from stalling the decode phase of other users currently in the batch.

Roofline Saturation and Aggregate Throughput

Continuous batching directly attacks the memory bandwidth bottleneck of the decode phase. By processing multiple requests at once, the GPU reads the massive model weights from High Bandwidth Memory (HBM) into the SM registers just one time, and uses those weights to compute tokens for every sequence in the batch.

Aggregate throughput scales non-linearly with the continuous batch size, eventually hitting a hardware memory bandwidth ceiling defined by the Roofline model:

$eff(B) = S \times (1 - e^{-B / S})$

In this formula, $B$ is the active batch size and $S$ is the hardware saturation threshold. The saturation threshold dictates how many concurrent requests are required to fully utilize the GPU's memory bandwidth.

On an enterprise A100 80GB accelerator with nearly 2.0 TB/s of memory bandwidth, you might need a concurrent batch size of 64 or 128 to reach maximum throughput efficiency. On Apple Silicon unified memory setups like an M2 Ultra 128GB with 800 GB/s bandwidth, saturation occurs at a much lower batch size of 16 to 32. Once you exceed the saturation threshold, adding more concurrent requests to the continuous batch will not increase overall tokens-per-second, but it will directly degrade the latency (TPOT) for every individual user because the fixed hardware resources are simply sliced thinner.

Mixture of Experts (MoE) Routing Constraints

Continuous batching behaves differently when sizing hardware for Mixture of Experts architectures like mistralai/Mixtral-8x7B-Instruct-v0.1. In a dense model, every token passes through every parameter. In an MoE model, the router network activates only a fraction of the parameters per token (e.g., 2 experts out of 8).

When continuous batch sizes are small, the batch might only touch a few experts, resulting in highly efficient memory reads. As you scale the continuous batch size up to 64 or 128 concurrent requests, probability dictates that the diverse tokens will route to all 8 experts simultaneously. This phenomenon is called expert thrashing. The GPU is suddenly forced to read the entire 47B parameter footprint from HBM in a single decode iteration rather than just the active 13B parameters. This drastically lowers the saturation threshold $S$, meaning MoE architectures hit their maximum throughput efficiency at much smaller continuous batch sizes than dense models of equivalent parameter counts. Infrastructure operators must provision smaller, tighter batch limits when deploying MoE models to avoid memory bandwidth stalling.