Speculative Decoding Systems

Autoregressive text generation fundamentally wastes hardware resources. During the decode phase, the GPU spends the majority of its time moving weights from High Bandwidth Memory (HBM) into the compute cores, rather than performing actual arithmetic. A modern 80GB accelerator like an H100 provides over 900 TFLOPs of compute but can only move data at 3.3 TB/s. Because standard decoding processes exactly one token at a time, the heavy computational units sit largely idle waiting for memory reads. Speculative decoding solves this memory bandwidth bottleneck by converting the sequential decode phase into a parallel prefill operation.

Instead of generating one token per forward pass, a speculative system uses a smaller, faster draft model to guess the next few tokens. The large target model then processes these guessed tokens all at once in a single batch. If the target model agrees with the draft tokens, it accepts them, effectively outputting multiple tokens in the time it usually takes to generate one.

Draft and Verify Mechanics

Speculative decoding operates on a two-step cycle. First, the draft model runs autoregressively to generate a sequence of $K$ tokens. Because the draft model is significantly smaller, it completes this task with very low latency and minimal compute cost. Next, the target model performs a single forward pass on all $K$ draft tokens concurrently. This verification step operates exactly like a prompt prefill phase. The target model computes the logits for the entire sequence in parallel, maximizing Tensor Core utilization.

The cyclic execution flow of speculative decoding, alternating between fast autoregressive drafting and parallel verification.

When the target model compares its generated probabilities against the draft tokens, it finds the first token where the predictions diverge. It accepts all tokens up to that point, discards the rest, and generates the correct token for the diverging step. Even if all $K$ tokens are rejected, the target model still guarantees the generation of at least one valid token per cycle.

Measuring Acceleration and Acceptance Rates

The performance gains depend entirely on the acceptance rate, denoted as $\alpha$. This represents the probability that the target model agrees with a given drafted token. The expected number of accepted tokens per verification step, $E$, is calculated using the geometric sum of the acceptance rate over the draft length $K$:

$E = \frac{1 - \alpha^{K+1}}{1 - \alpha}$

For example, in a production environment running meta-llama/Meta-Llama-3.1-70B-Instruct as the target and meta-llama/Meta-Llama-3.1-8B-Instruct as the draft model on an 8-GPU node. If you configure a draft length of $K=4$ and observe an average acceptance rate of $0.7$ for standard text generation, the expected tokens per step is:

$E = \frac{1 - 0.7^5}{1 - 0.7} = \frac{1 - 0.16807}{0.3} = 2.77 \text{ tokens}$

Without speculation, a single forward pass yields exactly 1 token. With speculation, a single forward pass yields an average of 2.77 tokens. Assuming the time taken by the small 8B model to generate 4 tokens plus the parallel verification pass of the 70B model is roughly equal to a single autoregressive step of the 70B model, the system achieves nearly a 2.7x speedup in generation latency.

Hardware and Memory Sizing Overheads

Colocating two models on the same GPU cluster introduces strict memory constraints. You must allocate VRAM for both sets of model weights and both KV caches. The total memory requirement follows this structure:

$Memory_{Total} = Memory_{Target Weights} + Memory_{Draft Weights} + Memory_{Target KV} + Memory_{Draft KV}$

Take a dual RTX 4090 24GB workstation (48GB VRAM total) running a quantized target model Qwen/Qwen2.5-32B-Instruct and a smaller draft model Qwen/Qwen2.5-1.5B-Instruct. If you load both models in 4-bit AWQ or GPTQ precision, you can calculate the exact weight footprints.

The 32B target model requires 16GB for weights. The 1.5B draft model requires roughly 0.75GB for weights. You must then size the KV cache for both. Since the draft model shares the exact same sequence length and batch size as the target, you calculate its KV cache footprint using the standard dimension equations. The 32B model has 32 layers, 8 KV heads, and a head dimension of 128. The 1.5B draft model has 28 layers, 2 KV heads, and a head dimension of 128.

If you run a batch size of 16 and a context window of 4096 tokens in FP16 precision (2 bytes per element), the target KV cache consumes:

$2 \times 32 \times 8 \times 128 \times 4096 \times 2 \times 16 = 8.58 \text{ GB}$

The draft KV cache footprint is:

$2 \times 28 \times 2 \times 128 \times 4096 \times 2 \times 16 = 0.37 \text{ GB}$

Total VRAM required is 16GB + 0.75GB + 8.58GB + 0.37GB = 25.7GB. This fits comfortably within the 48GB capacity of the dual GPU setup, leaving ample headroom for CUDA context buffers and continuous batching operations.

Implementation Architecture

Serving engines manage the request routing, draft token generation, and logits verification automatically. When defining the deployment configuration, you specify the target model, the draft model, and the maximum number of speculation steps ($K$).

Pairing a draft model with the target model sets up speculative verification during engine initialization:

from vllm import LLM, SamplingParams

target_model = "meta-llama/Meta-Llama-3.1-70B-Instruct"
draft_model = "meta-llama/Meta-Llama-3.1-8B-Instruct"

llm = LLM(
    model=target_model,
    tensor_parallel_size=4,
    speculative_model=draft_model,
    num_speculative_tokens=5,
    trust_remote_code=True
)

sampling_params = SamplingParams(temperature=0.0, max_tokens=256)
outputs = llm.generate(["Explain the mechanics of speculative decoding."], sampling_params)

for output in outputs:
    print(output.outputs[0].text)

In this configuration, vLLM allocates the target model across 4 GPUs using tensor parallelism. The engine automatically provisions the memory for both models and schedules the draft-verify loop. Setting num_speculative_tokens=5 assigns $K=5$, instructing the 8B model to generate up to 5 tokens ahead before triggering the 70B verification pass.

Draft-Free Alternative Architectures

Managing two separate models complicates deployment pipelines and doubles the artifact storage requirements. To circumvent this, infrastructure engineers often adopt multi-head architectural variants like Medusa or Eagle.

Instead of a separate draft model, these systems attach additional projection heads to the final layer of the target model. A standard language model has a single LM head that predicts token $t+1$. A Medusa architecture adds secondary heads that simultaneously predict tokens $t+2$, $t+3$, and $t+4$ using the same hidden state. Because these extra heads share the primary model parameters and KV cache, the weight memory overhead is strictly limited to the parameter count of the additional projection matrices. This unified approach eliminates the need to maintain a separate draft KV cache, freeing VRAM for higher batch concurrency.