Regularization Techniques: L1 and L2

As we saw earlier, overfitting occurs when a network learns the training data too well, including its noise and specific idiosyncrasies, failing to generalize to new data. One effective strategy to combat this is regularization, which involves adding a penalty to the loss function based on the magnitude of the network's weights. The intuition is that complex models often have large weights, while simpler, more generalizable models tend to have smaller weights. By penalizing large weights, we encourage the network to find simpler solutions.

Two common and powerful regularization techniques directly inspired by linear models are L1 and L2 regularization.

L2 Regularization (Weight Decay)

L2 regularization adds a penalty proportional to the square of the value of each weight ($w$) to the original loss function ($L_{original}$). The modified loss function, $L_{L2}$, becomes:

$$L_{L2} = L_{original} + \frac{\lambda}{2m} \sum_{i} w_i^2$$

Here:

• $w_i$ represents each weight in the network (biases are sometimes excluded).
• $\sum_{i} w_i^2$ is the sum of the squares of all weights.
• $\lambda$ (lambda) is the regularization hyperparameter, controlling the strength of the penalty. A higher $\lambda$ imposes a stronger penalty.
• $m$ is the number of examples in the batch (or dataset), used for scaling. The factor of $\frac{1}{2}$ is often included for mathematical convenience when calculating the gradient.

How it works: During backpropagation, the gradient of this penalty term with respect to a weight $w_i$ is $\frac{\lambda}{m} w_i$. When updating the weights using gradient descent, this term gets subtracted:

$$w_i := w_i - \alpha \left( \frac{\partial L_{original}}{\partial w_i} + \frac{\lambda}{m} w_i \right)$$

Notice the term $-\alpha \frac{\lambda}{m} w_i$. This term effectively pushes the weight $w_i$ slightly towards zero in each update step, proportional to its current value. This is why L2 regularization is often called Weight Decay. It encourages the network to use smaller weights, distributing the importance across many neurons rather than relying heavily on a few large weights. This generally leads to smoother decision boundaries and better generalization.

The quadratic penalty ($w^2$) imposed by L2 regularization. The penalty grows significantly as weights move away from zero, strongly discouraging large weight values.

L1 Regularization

L1 regularization takes a different approach. It adds a penalty proportional to the absolute value of each weight to the loss function:

$$L_{L1} = L_{original} + \frac{\lambda}{m} \sum_{i} |w_i|$$

Here, $\sum_{i} |w_i|$ is the sum of the absolute values of all weights, and $\lambda$ again controls the penalty strength.

How it works: The gradient of the L1 penalty term is $\frac{\lambda}{m} \text{sign}(w_i)$, where $\text{sign}(w_i)$ is +1 if $w_i$ is positive, -1 if $w_i$ is negative, and 0 if $w_i$ is zero (though the gradient is technically undefined at zero, implementations handle this). The weight update looks like:

$$w_i := w_i - \alpha \left( \frac{\partial L_{original}}{\partial w_i} + \frac{\lambda}{m} \text{sign}(w_i) \right)$$

The key difference from L2 is that L1 subtracts a constant factor ($\alpha \frac{\lambda}{m}$) pushing the weight towards zero, regardless of the weight's current magnitude (as long as it's not zero). This constant push can make weights become exactly zero and stay there. Consequently, L1 regularization often leads to sparse models, where many weights are zero. This can be interpreted as a form of automatic feature selection, as neurons connected by zero weights effectively become inactive for those inputs.

The linear penalty ($|w|$) imposed by L1 regularization. The penalty increases linearly as weights move away from zero. The constant gradient encourages sparsity by pushing small weights directly to zero.

Choosing Between L1 and L2

• L2 (Weight Decay) is generally more common and often performs well as a default. It encourages small, diffuse weights.
• L1 produces sparse weights, effectively performing feature selection by driving some weights to zero. It can be useful if you suspect many input features are irrelevant or if you desire a sparser, potentially more interpretable model. However, it can sometimes be less stable during training compared to L2.

In practice, both L1 and L2 regularization can be added directly within most deep learning frameworks when defining layers or optimizers.

The Regularization Parameter ($\lambda$)

The choice of the regularization hyperparameter $\lambda$ is important.

• If $\lambda = 0$, there is no regularization.
• If $\lambda$ is too small, regularization has little effect, and the model might still overfit.
• If $\lambda$ is too large, the penalty on weights dominates the loss. The network will prioritize minimizing weights over fitting the data, potentially leading to underfitting (high bias).

Finding a good value for $\lambda$ typically requires experimentation. It's usually tuned using the validation set, often through techniques like grid search or random search, which we will discuss later in this chapter under hyperparameter tuning.

By adding these penalties, L1 and L2 regularization provide effective mechanisms to control model complexity, discourage overfitting, and improve the generalization performance of your neural networks on unseen data. They are fundamental tools in the deep learning practitioner's toolkit.