Magnitude-Based Pruning

Magnitude-based pruning operates on a straightforward premise: model parameters, typically weights, with smaller absolute values contribute less significantly to the model's output and can therefore be removed with minimal impact on performance. This approach directly targets the creation of sparsity within the weight matrices of the model.

The Core Idea: Small Weights, Small Impact?

The intuition behind magnitude pruning stems from how neural networks compute outputs. In many operations, particularly matrix multiplications ($Y = WX + b$), the output $Y$ is a weighted sum of inputs $X$, where the weights are the elements of $W$. If a specific weight $w_{ij}$ has a magnitude close to zero, its contribution to the corresponding output element $y_i$ (specifically, the term $w_{ij} x_j$) will also be small, regardless of the input $x_j$. Removing such a weight (setting it to zero) is hypothesized to alter the output less dramatically than removing a weight with a large magnitude.

While this is a heuristic, it often holds surprisingly well in practice for overparameterized models like LLMs, where many weights might indeed be redundant or contribute marginally.

One-Shot Pruning: The Simplest Form

The most basic implementation is one-shot pruning. This involves:

1. Training: Train the dense LLM to convergence or start with a pre-trained model.
2. Ranking: Calculate the absolute magnitude $|w|$ for every weight $w$ in the target layers or the entire model.
3. Thresholding: Determine a pruning threshold. This can be a global threshold value or, more commonly, defined by a target sparsity level $S$. For a target sparsity $S$, the $(S \times 100)$-th percentile of weight magnitudes is found, and all weights with magnitudes below this value are set to zero.
4. Applying Mask: Create a binary mask $M$ where $M_{ij} = 0$ if $|w_{ij}|$ is below the threshold and $M_{ij} = 1$ otherwise. The pruned weights $W_{pruned}$ are then $W_{pruned} = W \odot M$, where $\odot$ denotes element-wise multiplication.

This approach is fast but can be overly aggressive. Removing a large fraction of weights simultaneously might significantly degrade model accuracy, sometimes irrecoverably.

Iterative Pruning: A More Gradual Approach

To mitigate the accuracy drop associated with one-shot pruning, iterative magnitude pruning (IMP) is commonly employed. Instead of removing all target weights at once, IMP follows a cycle:

1. Prune: Remove a small fraction of the currently active weights with the lowest magnitudes (e.g., 5-10% of the remaining weights).
2. Fine-tune: Retrain the pruned model for a limited number of epochs (fine-tuning) on the original training data or a relevant subset. This allows the remaining weights to adapt and compensate for the removed ones, recovering lost accuracy.
3. Repeat: Continue this prune-and-fine-tune cycle until the desired overall sparsity level is achieved.

Iterative Magnitude Pruning (IMP) process: Pruning small fractions of weights followed by fine-tuning cycles.

This gradual removal and interleaved fine-tuning generally lead to much better accuracy at higher sparsity levels compared to one-shot pruning.

Fine-tuning Schedules in IMP

The fine-tuning step is essential for the success of iterative pruning. Important considerations include:

• Learning Rate: Often, a smaller learning rate than the initial pre-training rate is used. This helps gently adjust the remaining weights without causing instability. Learning rate schedules (e.g., gradual decay) might also be beneficial.
• Duration: Fine-tuning typically requires fewer epochs than the original training. The goal is accuracy recovery, not training from scratch. The optimal number of epochs per cycle is often determined empirically.
• Masking: During fine-tuning, the gradients for the pruned weights (those set to zero) must remain zero. Only the unpruned weights should be updated. This is typically achieved by applying the pruning mask to the gradients before the optimizer step.

Determining Sparsity and Scope

A significant decision is the target sparsity level. Higher sparsity leads to smaller models and potentially faster inference (if supported by hardware/software), but usually comes at the cost of accuracy. The relationship is often non-linear: initial pruning might have little effect, but accuracy can drop sharply after a certain point.

Accuracy degradation as sparsity increases during iterative magnitude pruning. Performance often remains stable initially but can decrease significantly at higher sparsity levels.

Another choice is the scope of pruning:

• Global Pruning: Ranks all weights across the entire model and applies a single threshold.
• Layer-wise Pruning: Ranks and prunes weights independently for each layer, applying potentially different sparsity levels per layer. This can sometimes yield better results as it allows adapting sparsity to the sensitivity of different layers.

Strengths and Weaknesses

Advantages:

• Simplicity: The core concept is intuitive and relatively straightforward to implement, especially the one-shot version.
• Effectiveness: Can achieve substantial model size reduction often with manageable accuracy loss, particularly with iterative methods.
• Generality: Applicable to various model architectures and layers containing weight parameters.

Disadvantages:

• Unstructured Sparsity: Standard magnitude pruning typically results in irregular, fine-grained sparsity patterns (individual weights zeroed out). This often doesn't translate directly into significant latency improvements on general-purpose hardware (like GPUs or CPUs) without specialized libraries or hardware support designed for sparse computations. We will discuss structured pruning later, which addresses this.
• Computational Cost: Iterative pruning requires repeated fine-tuning cycles, which can be computationally expensive and time-consuming, especially for very large models.
• Magnitude ≠ Importance: The assumption that low magnitude always means low importance is a heuristic. Some low-magnitude weights might become important during fine-tuning, or certain structures might rely on combinations of small weights.

Magnitude-based pruning serves as a fundamental technique in the LLM optimization toolkit. While effective for reducing model size, its impact on inference speed is often indirect unless paired with specific hardware or software support. Understanding its principles and the iterative refinement process is essential before exploring more complex structured pruning methods or techniques that dynamically adjust sparsity during training.