Transformer Execution Lifecycle

Generating text with a Large Language Model is not a single atomic operation. It is a strictly ordered sequence of matrix multiplications, memory fetches, and vector transformations that execute repeatedly for every single word generated. To design efficient hardware infrastructure, you must understand the exact mechanical steps the GPU performs during this process. The transformer execution lifecycle dictates how data moves between the physical memory chips (HBM or GDDR) and the computational cores (Streaming Multiprocessors). If you miscalculate the memory required for any single step of this lifecycle, your model will either fail with an Out of Memory error or run at a fraction of the hardware capability.

To ground this lifecycle in reality, tracing the forward pass step-by-step with PyTorch shows how the GPU processes embeddings, transformer layers, and token selection sequentially:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load Llama-3.1-8B on an RTX 3090/4090 24GB or A100 80GB
model_id = "meta-llama/Meta-Llama-3.1-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    device_map="cuda"
)

input_text = "The physical hardware limits"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

with torch.no_grad():
    # 1. Embedding Lookup
    hidden_states = model.model.embed_tokens(input_ids)
    
    # 2. Transformer Block Loop (32 layers for Llama-3.1-8B)
    for layer in model.model.layers:
        hidden_states = layer(hidden_states)[0]
        
    # 3. Final Normalization
    hidden_states = model.model.norm(hidden_states)
    
    # 4. Logit Projection (LM Head)
    logits = model.lm_head(hidden_states)
    
    # 5. Token Selection
    next_token_id = torch.argmax(logits[:, -1, :], dim=-1)
    
print(f"Generated Token ID: {next_token_id.item()}")
print(f"Decoded: {tokenizer.decode(next_token_id)}")

The Input Pipeline Tokenization and Embeddings

The lifecycle begins outside the neural network. Neural networks cannot process raw strings. The tokenizer converts the raw input text into a sequence of integer IDs based on a pre-defined vocabulary. For meta-llama/Meta-Llama-3.1-8B, the vocabulary size is 128,256.

Once the integers are generated, they pass to the embedding layer. The embedding layer is a large lookup table resident in the GPU memory. It maps each integer ID to a dense, continuous vector representation. The dimensions of this matrix are the vocabulary size multiplied by the model hidden dimension.

The memory footprint of this single layer is significant. For the 8B model with a hidden dimension of 4096 loaded in 16-bit precision (FP16, which requires 2 bytes per parameter), we can calculate the exact VRAM footprint of the embedding matrix.

$Memory_{embedding} = vocab\_size \times hidden\_dim \times bytes\_per\_element$

For meta-llama/Meta-Llama-3.1-8B: $128,256 \times 4096 \times 2 = 1,050,673,152 \text{ bytes} \approx 1.05 \text{ GB}$

Every time a request starts, the GPU allocates a runtime activation tensor to hold the embedded representation of your input sequence. If your batch size is 1 and your sequence length is 2048 tokens, the activation tensor shape is [1, 2048, 4096]. At 2 bytes per element, this tensor consumes 16.7 MB of VRAM.

The Transformer Block Loop Attention and Feed Forward

After the embedding lookup, the activation tensor enters the main execution loop. The core of any Large Language Model is a stack of identical transformer blocks. A dense 8B model typically contains 32 layers. A 70B model like meta-llama/Meta-Llama-3.1-70B-Instruct contains 80 layers.

Inside each block, the execution splits into two primary computational phases.

First is the Self-Attention mechanism. The activation tensor is multiplied by three separate weight matrices to create the Query, Key, and Value (QKV) projections. The attention mechanism allows every token in the sequence to exchange information with preceding tokens. The resulting context vectors are then multiplied by an output projection matrix.

Second is the Feed-Forward Network (MLP). The context-aware tensor passes through a series of linear layers that expand the hidden dimension and then project it back down. In many modern architectures using SwiGLU activations, the hidden dimension expands by a factor of roughly 3.5 to 4. For an 8B model with a 4096 hidden dimension, the intermediate MLP dimension is typically 14336.

The execution flow of a single inference step. The generation of one new token requires passing the input sequence through every layer sequentially. The output is then appended to the sequence for the next execution cycle.

From a hardware perspective, these layers are strictly sequential. The GPU cannot compute Layer 2 until Layer 1 finishes. During token generation, the GPU must load the complete set of weights for Layer 1 from VRAM into the Streaming Multiprocessors, perform matrix operations against the small activation tensor, and write the result back to VRAM. It must repeat this data transfer for all 32 layers to generate a single word.

Output Projection and Token Selection

Once the activation tensor exits the final transformer block, it undergoes a final layer normalization. The tensor then passes through the Language Model Head (LM Head). This is a linear projection matrix that maps the hidden dimension back to the vocabulary size.

The LM Head has the exact same dimensions as the embedding layer: hidden_dim x vocab_size. The resulting tensor contains the raw, unnormalized scores (logits) for every possible token in the vocabulary.

The final execution step is token selection. The system applies a softmax function to convert the logits into a probability distribution. Depending on your configuration parameters, the system will either pick the token with the highest probability (greedy decoding) or sample from the distribution based on temperature settings. The chosen token is returned to the user, appended to the input sequence, and the entire execution lifecycle starts over from the very beginning.

Sizing Runtime Activations

While calculating static model weights is straightforward, understanding the dynamic memory footprint of the activation tensor is critical for infrastructure planning. During inference, this tensor constantly changes size based on the batch size and sequence length.

The memory required to store a single hidden state activation tensor in VRAM is defined by the following equation.

$Memory_{activation} = batch\_size \times seq\_length \times hidden\_dim \times bytes\_per\_element$

For example, in an enterprise deployment running Qwen/Qwen2.5-32B on dual RTX 4090 24GB GPUs (48GB total VRAM), the 32B model has a hidden dimension of 5120. If you are processing a document with a sequence length of 32,768 tokens in a batch of 1 using FP16 precision, the activation tensor size is calculated directly.

$1 \times 32,768 \times 5120 \times 2 = 335,544,320 \text{ bytes} \approx 335.5 \text{ MB}$

While 335 MB seems small relative to 48 GB of total VRAM, this is just a single tensor. During the forward pass, operations like the attention matrix multiplication generate multiple intermediate tensors of size [batch_size, num_heads, seq_length, seq_length]. For long context windows, these intermediate attention matrices can easily spike to several gigabytes of VRAM allocation, dictating the maximum batch size your hardware can support before crashing.