Defining Latency SLAs TTFT TPOT and QPS

Serving Large Language Models in production requires translating raw hardware specifications into predictable user experiences. When an application requests text generation, the underlying compute infrastructure must execute billions of matrix multiplications within strict time bounds. To design a cluster that satisfies user demand without over-provisioning expensive accelerators, infrastructure operators rely on three primary service level agreement (SLA) metrics: Time-To-First-Token (TTFT), Time-Per-Output-Token (TPOT), and Queries Per Second (QPS). Mapping these metrics directly to GPU memory bandwidth and compute throughput dictates the physical hardware required.

Request execution phases mapped to latency metrics.

Time-Per-Output-Token (TPOT) and Memory Bandwidth

TPOT defines the delay between each generated token during the autoregressive decode phase. Because text generation produces one token at a time, the GPU must load the entire model's weights from High Bandwidth Memory (HBM) into the compute cores (SRAM) for every single generation step. This makes TPOT strictly bottlenecked by the physical memory bandwidth of the GPU, not its raw computational speed.

To calculate the baseline TPOT for a batch size of one, divide the total memory footprint of the active model parameters by the effective memory bandwidth of the hardware:

$TPOT_{ideal} = \frac{Active\_Parameters \times Bytes\_Per\_Parameter}{Memory\_Bandwidth}$

For example, running the 8-billion parameter meta-llama/Meta-Llama-3.1-8B at 16-bit precision (FP16 or BF16) on a single NVIDIA RTX 3090. In FP16, each parameter occupies 2 bytes, meaning the active parameters require 16 GB of memory. The RTX 3090 has a theoretical memory bandwidth of 936 GB/s.

$TPOT_{RTX3090} = \frac{8,000,000,000 \times 2 \text{ bytes}}{936 \text{ GB/s}} = \frac{16 \text{ GB}}{936 \text{ GB/s}} \approx 0.017 \text{ seconds (17 ms)}$

A TPOT of 17 milliseconds translates to roughly 58 tokens per second. Human reading speed averages 4 to 5 words per second, or roughly 6 to 8 tokens per second. Therefore, an acceptable SLA for streaming text requires a TPOT below 125 ms. The single RTX 3090 easily satisfies this single-user SLA.

Scaling up to meta-llama/Meta-Llama-3.1-70B-Instruct in FP16 requires 140 GB of parameter memory. This model will not fit on a single RTX 3090 or a single 80 GB A100. Deploying it across two A100 80GB GPUs provides a combined 160 GB of VRAM. The A100 features a memory bandwidth of 2,039 GB/s. Assuming optimal Tensor Parallelism over NVLink:

$TPOT_{Dual\_A100} = \frac{140 \text{ GB}}{2 \times 2,039 \text{ GB/s}} \approx 0.034 \text{ seconds (34 ms)}$

For Mixture of Experts (MoE) architectures like mistralai/Mixtral-8x7B-Instruct-v0.1, the calculation uses only the active parameters. While the model has 47B total parameters (requiring 94 GB of VRAM in FP16), only 13B parameters activate per token. This sparsity keeps TPOT low. It requires only the bandwidth to load 26 GB of data per step, even though the total capacity footprint is much larger.

Time-To-First-Token (TTFT) and Compute Limits

TTFT measures the time elapsed from when the user submits a prompt to when the system returns the first generated token. This metric is governed by the prefill phase, where the model processes all input tokens simultaneously in massive parallel matrix multiplications. Unlike the decode phase, the prefill phase is typically bound by the arithmetic logic units (Tensor Cores), measured in TeraFLOPs (TFLOPs).

The total floating-point operations (FLOPs) required to prefill a prompt can be estimated using the forward-pass approximation:

$Prefill\_FLOPs = 2 \times Active\_Parameters \times Prompt\_Token\_Count$

If a user submits a 2,048-token context window to Qwen/Qwen2.5-32B (32 billion parameters), the required FLOPs scale predictably:

$Prefill\_FLOPs = 2 \times 32,000,000,000 \times 2048 \approx 131 \text{ TeraFLOPs}$

A single A100 80GB GPU delivers approximately 312 TFLOPs of dense FP16 compute. At an idealized 100% utilization, the physical compute time evaluates to:

$TTFT = \frac{131 \text{ TFLOPs}}{312 \text{ TFLOPs/second}} \approx 0.42 \text{ seconds (420 ms)}$

Interactive chatbots typically enforce a TTFT SLA of under 1.0 second to prevent users from abandoning the session. Code completion agents require stricter SLAs, often demanding TTFT under 250 ms to feel instantaneous while typing. As sequence lengths grow to 32k or 128k context windows, TTFT calculations must also account for the quadratic scaling of attention mechanics, which eventually overtakes the standard parameter-FLOP calculation shown above.

Queries Per Second (QPS) and Roofline Saturation

While TTFT and TPOT represent the latency experienced by a single user, Queries Per Second (QPS) measures the aggregate throughput of the deployment. To increase QPS, inference engines batch multiple requests together.

Batching amortizes the memory bandwidth cost. When processing a batch size of 8, the GPU loads the 16 GB of meta-llama/Meta-Llama-3.1-8B parameters once but multiplies them against 8 distinct token states. This increases overall throughput (total tokens per second) at the expense of a slight increase in latency (TPOT) due to larger matrix-vector sizes and KV cache memory constraints.

Throughput follows a saturation curve. Adding concurrent requests increases QPS up to a threshold defined by the hardware's maximum memory bandwidth and compute limit.

Roofline saturation demonstrating how increasing batch concurrency improves QPS until memory bandwidth maxes out, after which TPOT SLA degrades rapidly.

Profiling SLAs in Production Code

To properly map required SLAs to hardware, benchmarking is mandatory. Open-source inference engines like vLLM manage dynamic batching and PagedAttention automatically, allowing operators to measure TTFT and TPOT under simulated load.

Spinning up an offline engine with vLLM enables benchmarking exact prefill and decode timings to extract baseline SLA metrics on local hardware:

import time
from vllm import LLM, SamplingParams

# Initialize the model on available GPU hardware
# Enforcing max_model_len to constrain KV cache sizing
model_id = "Qwen/Qwen2.5-7B"
llm = LLM(model=model_id, max_model_len=2048, dtype="bfloat16")

# Define generation parameters for evaluation
sampling_params = SamplingParams(temperature=0.0, max_tokens=100)
prompt = "Explain the architecture of a transformer model in extensive detail."

# Warm-up run to initialize CUDA graphs and allocate static memory
llm.generate([prompt], sampling_params, use_tqdm=False)

# Measure TTFT (prefill) and TPOT (decode) phases
start_time = time.perf_counter()

# Generate response
outputs = llm.generate([prompt], sampling_params, use_tqdm=False)
end_time = time.perf_counter()

# Calculate metrics
total_time = end_time - start_time
output_tokens = len(outputs[0].outputs[0].token_ids)

# Precise TTFT requires intercepting the stream natively
# Average TPOT is derived from total runtime minus estimated prefill
avg_tpot = (total_time / output_tokens) * 1000

print(f"Hardware Profiling for {model_id}")
print(f"Generated {output_tokens} tokens in {total_time:.2f} seconds.")
print(f"Average Inter-Token Latency (TPOT): {avg_tpot:.2f} ms")

By altering the list of prompts passed to the engine, you can simulate concurrent load. Monitoring these metrics against physical hardware limits allows you to confidently size production clusters. If TPOT crosses your SLA threshold (for example, exceeding 100 ms), the node has reached its batching limit, and additional QPS demand must be routed to a new replicated instance, scaling the cluster horizontally.