Prefill Phase Compute Operations

When a Large Language Model receives a prompt, it does not immediately begin generating text. Before outputting the first word, the model must process the entire input sequence to build contextual representations and populate the initial Key-Value (KV) cache. This initial processing stage is the prefill phase. Because the entire prompt is available at once, the system can process all tokens in parallel using massive General Matrix Multiply (GEMM) operations. This parallel execution fundamentally defines the hardware requirements for prompt processing, making it heavily dependent on the raw arithmetic throughput of the GPU rather than its memory bandwidth.

Arithmetic Intensity in Prompt Processing

To evaluate hardware bottlenecks, infrastructure engineers measure a workload's arithmetic intensity. This metric defines the ratio of compute operations performed relative to the amount of data transferred from memory.

$Arithmetic\_Intensity = \frac{Total\_FLOPs}{Total\_Memory\_Bytes\_Accessed}$

During the prefill phase, the GPU loads the model weights into its compute cores once, and then multiplies those identical weights across all $N$ tokens in the input prompt simultaneously. If a model is loaded in 16-bit precision (FP16 or BF16), each parameter requires 2 bytes of memory. Processing a prompt requires approximately 2 FLOPs per parameter per token (one multiply and one add operation).

By simplifying the equation for the prefill phase, the relationship between prompt length and arithmetic intensity becomes clear:

$Arithmetic\_Intensity = \frac{2 \times Parameters \times N}{Parameters \times 2 \text{ bytes}} = N \text{ FLOPs per byte}$

If you send a 4,096-token prompt to the model, the prefill arithmetic intensity is 4,096 FLOPs per byte. Modern enterprise accelerators like the NVIDIA A100 (80GB SXM4) provide about 312 TFLOPs of FP16 compute and 2,039 GB/s of memory bandwidth, resulting in a hardware balance of approximately 153 FLOPs per byte. Because the workload's intensity (4,096) far exceeds the hardware's balance (153), the GPU's memory bandwidth is saturated with compute. The Tensor Cores operate at maximum capacity, making the operation strictly compute-bound.

The prefill execution flow prioritizes parallel matrix multiplication, isolating the compute-heavy prompt processing from the subsequent memory-bound token generation.

Calculating Prefill Compute Requirements

You can calculate the exact number of floating-point operations required to process a prompt using the basic FLOPs heuristic.

$FLOPs_{prefill} = 2 \times P \times N$

For example, deploying meta-llama/Meta-Llama-3.1-8B on a single NVIDIA RTX 4090 (24GB). You receive an incoming request with a 4,096-token prompt.

Using the formula:

• $P = 8,000,000,000$ (8B parameters)
• $N = 4,096$ (tokens)
• $FLOPs_{prefill} = 2 \times 8,000,000,000 \times 4,096 = 65,536,000,000,000 \text{ FLOPs}$ (65.5 TFLOPs)

The RTX 4090 offers a theoretical peak of 330 TFLOPs in FP16. However, highly optimized inference engines rarely achieve 100% Model FLOPs Utilization (MFU). Assuming a realistic MFU of 60%, the effective compute throughput is roughly 198 TFLOPs.

To find the theoretical compute time: $Time = \frac{65.5 \text{ TFLOPs}}{198 \text{ TFLOPs}} \approx 0.33 \text{ seconds}$

This duration is known as Time-to-First-Token (TTFT). For a user interacting with this 8B model, they will wait approximately 330 milliseconds before the system finishes the prefill computations and begins streaming the response.

Measuring Time-to-First-Token (TTFT)

To isolate and profile the prefill phase, setting the generation limit to a single token measures the exact TTFT overhead for a given prompt length:

from vllm import LLM, SamplingParams
import time

# Initialize the model on a single GPU
llm = LLM(model="meta-llama/Meta-Llama-3.1-8B", tensor_parallel_size=1)

# Create a sample prompt of substantial length
prompt = "Explain the hardware architecture of transformer models. " * 500 
prompts = [prompt]

# Isolate prefill by limiting generation to a single token
sampling_params = SamplingParams(max_tokens=1)

start_time = time.time()
outputs = llm.generate(prompts, sampling_params)
ttft = time.time() - start_time

print(f"Prompt length: {len(outputs[0].prompt_token_ids)} tokens")
print(f"Time to First Token (TTFT): {ttft:.4f} seconds")

Implications for Mixture of Experts (MoE)

When sizing hardware for Mixture of Experts architectures, the compute operations during the prefill phase differ from dense models. In an MoE model, only a subset of parameters are active for any given token.

Take mistralai/Mixtral-8x7B-Instruct-v0.1 as an example. While the model requires roughly 47 billion parameters to be resident in GPU memory (demanding a multi-GPU setup like dual RTX 6000 Ada 48GB cards or a single A100 80GB), only about 13 billion parameters are executed per token during the forward pass.

For a 4,096-token prompt, the prefill FLOP calculation uses the active parameter count rather than the total parameter count:

$FLOPs_{prefill} = 2 \times 13,000,000,000 \times 4,096 \approx 106.5 \text{ TFLOPs}$

While compute demands are reduced relative to a dense 47B model, the prefill phase for MoE models remains strictly compute-bound. The routing mechanisms assign different tokens in the prompt sequence to different expert layers simultaneously. Because the entire prompt is evaluated in parallel, the GPU can efficiently batch operations for each expert, maintaining high Tensor Core utilization before transitioning into the memory-bandwidth constrained decode phase.