L1 and L2 Regularization

As introduced earlier, overfitting occurs when a model learns the training data too well, capturing noise and specific patterns that don't generalize to new data. A common strategy to combat this is regularization, which involves adding a penalty to the model's loss function based on the complexity of the model itself. L1 and L2 regularization are two widely used techniques that penalize model complexity based on the magnitude of the network's weights.

The core idea is that models with excessively large weights might be too sensitive to small changes in the input features, essentially memorizing the training data. By adding a penalty term to the loss function that increases with the size of the weights, we encourage the optimization process (like gradient descent) to find solutions that not only minimize the original prediction error but also keep the weights relatively small.

L2 Regularization (Weight Decay or Ridge)

L2 regularization adds a penalty proportional to the square of the magnitude of the weights. The modified loss function looks like this:

$$\text{New Loss} = \text{Original Loss} + \lambda \sum_{i} w_i^2$$

Here:

• $w_i$ represents each weight in the network.
• $\sum_{i} w_i^2$ is the sum of the squares of all weights (the squared L2 norm).
• $\lambda$ (lambda) is the regularization strength hyperparameter. It's a positive value that you choose before training.

How it Works: The L2 penalty term $\lambda \sum w_i^2$ discourages large individual weights. During backpropagation, the gradient calculation for each weight $w_i$ will include an additional term proportional to $w_i$ itself ($2 \lambda w_i$). This means that during the weight update step of gradient descent ($w_i \leftarrow w_i - \text{learning\_rate} \times \text{gradient}$), there's an extra push pulling the weight towards zero. This effect is why L2 regularization is often called weight decay.

Effect:

• Smaller, Diffuse Weights: It encourages the network to use all its inputs to some extent, rather than relying heavily on just a few potentially noisy inputs. Weights tend to be distributed more evenly and take on smaller values.
• Smoother Models: Models with smaller weights tend to be less complex and have smoother decision boundaries, making them less prone to overfitting.
• Non-Sparse: L2 regularization drives weights towards zero but rarely makes them exactly zero.

The Role of λ: The hyperparameter $\lambda$ controls the trade-off between minimizing the original loss (fitting the data well) and minimizing the weight magnitudes (keeping the model simple).

• If $\lambda = 0$, there's no regularization.
• If $\lambda$ is very small, the regularization effect is minimal.
• If $\lambda$ is large, the weights will be heavily penalized, potentially leading to underfitting if the penalty dominates the original loss.

Choosing the right value for $\lambda$ typically involves experimentation and techniques like cross-validation.

L2 regularization tends to cluster weights closer to zero compared to an unregularized model.

L1 Regularization (Lasso)

L1 regularization adds a penalty proportional to the absolute value of the weights:

$$\text{New Loss} = \text{Original Loss} + \lambda \sum_{i} |w_i|$$

Here:

• $|w_i|$ is the absolute value of weight $w_i$.
• $\sum_{i} |w_i|$ is the sum of the absolute values of all weights (the L1 norm).
• $\lambda$ is again the regularization strength hyperparameter.

How it Works: The L1 penalty $\lambda \sum |w_i|$ also discourages large weights. However, the gradient contribution from the L1 term is proportional to the sign of the weight ($\lambda \times \text{sign}(w_i)$), assuming $w_i \neq 0$. This means that during optimization, each weight is pushed towards zero by a constant amount (determined by $\lambda$ and the learning rate), regardless of its current magnitude (unlike L2 where the push decreases as the weight approaches zero).

Effect:

• Sparsity: This constant push towards zero makes L1 regularization very effective at driving some weights to become exactly zero.
• Feature Selection: When a weight connected to an input feature becomes zero, it means that feature is effectively ignored by the model. Therefore, L1 can perform automatic feature selection, identifying and discarding less relevant inputs.
• Simpler Models: By zeroing out weights, L1 can lead to simpler, sparser models.

Shapes of L1 (diamond) and L2 (circle) constraint regions. Optimization finds a point where the loss function contour touches the constraint. L1's sharp corners make intersections along the axes (where one weight is zero) more probable.

The Role of λ: Similar to L2, $\lambda$ in L1 controls the strength of the regularization. A larger $\lambda$ leads to more weights being driven to zero, resulting in a sparser model.

Choosing Between L1 and L2

• L2 (Weight Decay) is generally the more common and often recommended starting point. It provides good generalization improvement by keeping weights small and preventing extreme values. Its smooth nature makes optimization slightly easier compared to L1.
• L1 (Lasso) is useful when you suspect many input features might be irrelevant and you desire a sparse model that performs automatic feature selection. However, it can sometimes be less stable during training due to the non-differentiable nature of the absolute value function at zero (though this is handled in practice). It might also discard features that are actually useful if they are highly correlated with other features.

In some cases, Elastic Net regularization, which combines both L1 and L2 penalties, is used to get benefits from both approaches.

Implementation Notes

Most deep learning frameworks make adding L1 or L2 regularization straightforward. For L2 (weight decay), it's often a simple parameter in the optimizer:

# Example in PyTorch using AdamW optimizer for L2 regularization (weight decay)
import torch
import torch.optim as optim

# Assume 'model' is your defined neural network
# learning_rate = 1e-3
# regularization_strength = 1e-4 # This is lambda for L2

# AdamW optimizer incorporates weight decay directly
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)

# --- Training loop would go here ---
# loss.backward()
# optimizer.step()
# ...

For L1, or sometimes for more granular control over L2, you might add the penalty term directly to your calculated loss before calling loss.backward().

Regularization techniques like L1 and L2 are important tools for preventing overfitting. By adding a penalty based on weight magnitudes to the loss function, they encourage simpler models that often generalize better to unseen data. Remember that the regularization strength $\lambda$ is a hyperparameter that usually requires tuning for optimal performance.