LoRA Memory Allocation

Fine-tuning large language models by updating every single parameter requires an immense amount of GPU memory. As you saw in previous calculations for full training, you must hold the model weights, the forward pass activations, the computed gradients, and the massive optimizer states in VRAM simultaneously. Low-Rank Adaptation minimizes this footprint by freezing the original pre-trained weights and appending small, trainable matrix pairs to specific layers in the network. Because you only train these newly injected matrices, you bypass the need to store gradients and optimizer states for the hundreds of billions of baseline parameters. Understanding exactly how these low-rank matrices allocate memory allows you to size your hardware precisely for fine-tuning workloads on infrastructure ranging from consumer GPUs to enterprise accelerators.

The Mechanics of Low Rank Adaptation

When a transformer processes tokens, the hidden states pass through numerous linear transformations inside the multi-head attention and feed-forward blocks. LoRA modifies these dense layers by adding a parallel computational path. The original weight matrix remains completely frozen in memory. Alongside it, LoRA injects two smaller matrices, conventionally named $A$ and $B$, which multiply together to approximate the weight updates.

If the original weight matrix has dimensions $d \times d$, the injected matrix $A$ will have dimensions $d \times r$, and matrix $B$ will have dimensions $r \times d$. The variable $r$ represents the rank.

$W_{new} = W_{base} + \frac{\alpha}{r} (B \times A)$

In this equation, $\alpha$ is a constant scaling factor that dictates the magnitude of the injected updates. Because $r$ is significantly smaller than the model dimension $d$, the total number of trainable parameters in matrices $A$ and $B$ is a tiny fraction of the base weight matrix.

The parallel execution path of a LoRA adapter. The massive base matrix remains frozen, while only the low-rank bottleneck matrices consume gradient and optimizer state memory during the backward pass.

Sizing the Trainable Parameters

To calculate the exact VRAM footprint added by LoRA, you first determine the raw number of trainable parameters. For a single square linear layer of dimension $d_{model}$, the trainable parameter count for the adapter is calculated as:

$Params_{LoRA} = 2 \times r \times d_{model}$

When planning your training cluster, you must decide which layers inside the transformer architecture will receive these adapters. Early implementations of LoRA only targeted the query and value projection layers (q_proj and v_proj) inside the attention mechanism. Modern best practices target all linear layers to achieve performance closer to full fine-tuning. This includes the attention projections (q_proj, k_proj, v_proj, o_proj) and the feed-forward network projections (gate_proj, up_proj, down_proj).

If you apply a rank $r = 16$ adapter to all linear layers of a standard 8B dense model like meta-llama/Meta-Llama-3.1-8B, the total trainable parameter count scales to approximately 21 million parameters. If you apply the exact same rank to a 70B model like meta-llama/Meta-Llama-3.1-70B-Instruct, the total trainable count reaches roughly 150 million parameters due to the larger hidden dimensions and layer counts.

Calculating Optimizer and Gradient Memory

Once you know the number of trainable parameters, you can calculate the exact memory required for the training states. While the frozen base model weights usually reside in 16-bit BF16 (consuming 2 bytes per parameter), the trainable LoRA adapters and their associated optimizer states require higher precision to maintain stability during gradient updates.

Using the standard AdamW optimizer, each trainable parameter requires:

1. The adapter weights themselves, usually cast to FP32 (4 bytes).
2. The gradients computed during the backward pass, in FP32 (4 bytes).
3. The optimizer's first moment estimate (4 bytes).
4. The optimizer's second moment estimate (4 bytes).

This results in a strict multiplier of 16 bytes of VRAM required for every single trainable LoRA parameter.

$Memory_{Adapter} = Params_{LoRA} \times 16 \text{ bytes}$

Applying this formula to our previous examples provides concrete hardware requirements. For the 8B model with 21 million trainable parameters, the entire training state overhead for LoRA is just 336 MB. For the 70B model with 150 million parameters, the adapter overhead is 2.4 GB. This is a massive reduction from the 1.1 TB of optimizer state memory required to fully fine-tune a 70B architecture.

Implementation and Targeting in PyTorch

Targeting linear projection layers with LoRA adapters reduces trainable parameters and optimizer state memory:

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

# Load the base model in 16-bit precision
model_id = "Qwen/Qwen2.5-7B"
base_model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto"
)

# Configure the LoRA adapter for all linear layers
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Inject the adapters into the base model
peft_model = get_peft_model(base_model, lora_config)
peft_model.print_trainable_parameters()

When you execute this configuration on Qwen/Qwen2.5-7B, the script will print that less than 0.5% of the total model parameters are trainable. This physical separation is what allows the adapter weights to be saved as an independent, lightweight file (often under 100 MB) rather than duplicating the entire 14 GB base model binary.

Adding Forward Activations and Batch Sizing

Calculating the base weights and the adapter optimizer states gives you the static VRAM requirement. To finalize your capacity planning, you must add the dynamic activation memory required for the forward pass. Unlike optimizer states, activation memory scales linearly with your batch size and sequence length.

When you pass a batch of tokens through the model, the framework must store the intermediate tensor outputs of every single layer to compute the chain rule during the backward pass. For a 32B model like Qwen/Qwen2.5-32B processing a sequence length of 4096 tokens with a batch size of 4, the raw activation memory can easily exceed 40 GB, immediately crashing a single consumer GPU even if the model weights fit perfectly.

To resolve this bottleneck, you apply Gradient Checkpointing. This technique drops the intermediate activations from VRAM and recomputes them on the fly during the backward pass. Gradient checkpointing trades compute for memory, increasing training time by roughly 20% to 30%, but reducing the dynamic activation footprint by a factor of the square root of the total layers.

Physical Cluster Mapping

By combining these sizing formulas, you can map specific workloads to physical hardware setups.

For an 8B model fine-tuning job, the base BF16 weights consume 16 GB. The LoRA training states consume roughly 0.4 GB. With gradient checkpointing and a conservative batch size, the activation memory requires 4 to 6 GB. This brings the total VRAM footprint to approximately 21 GB, allowing the entire job to fit comfortably inside a single RTX 3090 or RTX 4090 24GB GPU.

For a 70B model, the base BF16 weights consume 140 GB. The LoRA training states consume 2.4 GB. Activation memory will add roughly 10 to 15 GB depending on context length. This 157 GB footprint cannot fit on a single device. You must distribute this workload across an enterprise setup using two 80GB A100 or H100 accelerators (160GB total VRAM) or utilize a high-capacity unified memory architecture like an Apple Silicon Mac Studio with 192GB of RAM. If you are constrained to dual RTX 4090 24GB cards (48GB total), you must use standard LoRA and apply quantization techniques like QLoRA to compress the 140 GB base model down to 4-bit precision, reducing the base footprint to 35 GB.