L2 Regularization (Weight Decay): Mechanism

Models can sometimes fit the training data too well, capturing noise and specific patterns that do not exist in the general dataset. This phenomenon, known as overfitting, significantly impairs a model's ability to generalize to new, unseen data. Weight regularization offers a direct way to combat this by controlling the complexity of the model itself. The intuition is straightforward: overly complex models often rely on very large weight values to precisely fit the training examples, including the noise. By discouraging large weights, we encourage simpler models.

L2 regularization, perhaps the most common form of weight regularization, implements this idea by adding a penalty to the model's loss function. This penalty is proportional to the sum of the squares of all the weights in the network. It's also frequently referred to as Weight Decay.

How L2 Regularization Works

Let's consider a standard loss function, like Mean Squared Error (MSE) or Cross-Entropy, which we'll denote as $L_{original}(\mathbf{W})$, where $\mathbf{W}$ represents all the weights in our network. L2 regularization modifies this objective by adding a penalty term:

$$L_{total}(\mathbf{W}) = L_{original}(\mathbf{W}) + \frac{\lambda}{2} \sum_{i} w_i^2$$

Here:

• $w_i$ represents an individual weight in the network.
• $\sum_{i} w_i^2$ is the sum of the squares of all weights. This is the squared $L_2$ norm of the weight vector, often written as $||\mathbf{W}||_2^2$.
• $\lambda$ (lambda) is a hyperparameter that controls the strength of the regularization. It determines how much we penalize large weights. A higher $\lambda$ means stronger regularization. The factor of $\frac{1}{2}$ is included mostly for mathematical convenience, simplifying the derivative.

The Impact on Training: Weight Decay

How does this new loss function change the training process? During backpropagation, we calculate gradients of the loss function with respect to the weights to update them using an optimizer like SGD. The gradient of the total loss $L_{total}$ with respect to a specific weight $w_j$ now includes an extra term derived from the regularization penalty:

$$\frac{\partial L_{total}}{\partial w_j} = \frac{\partial L_{original}}{\partial w_j} + \lambda w_j$$

Look closely at the second term, $\lambda w_j$. When we update the weight $w_j$ using gradient descent with learning rate $\eta$ (eta), the update rule becomes:

$$w_j \leftarrow w_j - \eta \left( \frac{\partial L_{original}}{\partial w_j} + \lambda w_j \right)$$

Rearranging this slightly gives:

$$w_j \leftarrow w_j - \eta \frac{\partial L_{original}}{\partial w_j} - \eta \lambda w_j$$ $$w_j \leftarrow (1 - \eta \lambda) w_j - \eta \frac{\partial L_{original}}{\partial w_j}$$

This final form reveals why L2 regularization is called Weight Decay. In each update step, before subtracting the original gradient component ($\eta \frac{\partial L_{original}}{\partial w_j}$), the weight $w_j$ is multiplied by $(1 - \eta \lambda)$. Since both the learning rate $\eta$ and the regularization strength $\lambda$ are positive, this factor is slightly less than 1. Therefore, each weight is effectively shrunk or decayed towards zero during every update, in addition to being adjusted based on the original loss gradient.

The Effect: Smoother Models

The practical consequence of this weight decay is that the optimization process favors solutions where weight magnitudes are smaller and more evenly distributed. It discourages any single weight from growing excessively large. Models with smaller weights tend to exhibit smoother behavior. Imagine fitting a curve to data points; a model with large weights might produce a very "wiggly" curve that hits every training point perfectly but behaves erratically elsewhere. A model trained with L2 regularization would prefer a smoother curve that captures the general trend, even if it doesn't fit every training point exactly. This smoothness often translates directly to better generalization on unseen data.

The strength of this effect is controlled by the hyperparameter $\lambda$.

• If $\lambda = 0$, there's no regularization, and we revert to the original loss.
• If $\lambda$ is very large, the penalty dominates, forcing weights very close to zero, potentially leading to underfitting (the model becomes too simple).
• Choosing an appropriate $\lambda$ (often through experimentation or techniques like cross-validation, covered later) is important for balancing the trade-off between fitting the training data well and keeping the model simple enough to generalize.

In summary, L2 regularization (Weight Decay) modifies the loss function to penalize large squared weights. This alters the gradient update rule, causing weights to shrink towards zero during training, leading to simpler, smoother models less prone to overfitting.