L2 Regularization: Mathematical Formulation

L2 regularization works by penalizing large weights to encourage simpler models and improve generalization. Mathematically, L2 regularization is formalized by modifying the model's objective function—the function minimized during training—through the addition of a term that represents the penalty for large weights.

The Regularized Loss Function

Typically, when training a neural network, we minimize a loss function, often denoted as $J(\theta)$, which measures the discrepancy between the model's predictions and the true target values. Common examples include Mean Squared Error (MSE) for regression or Cross-Entropy Loss for classification. Here, $\theta$ represents all the learnable parameters (weights and biases) in the network.

L2 regularization adds a penalty term to this original loss function. This penalty is proportional to the squared magnitude of the weights. For a network with weight matrices $W^{[1]}, W^{[2]}, ..., W^{[L]}$ for layers $1$ to $L$, the L2 regularization term is calculated as the sum of the squared Frobenius norms of these weight matrices:

$$\text{L2 Penalty} = \frac{\lambda}{2m} \sum_{l=1}^{L} ||W^{[l]}||_F^2$$

Let's break this down:

• $W^{[l]}$ is the weight matrix for layer $l$.
• $||W^{[l]}||_F^2$ is the squared Frobenius norm of the weight matrix $W^{[l]}$. The Frobenius norm of a matrix is computed by squaring every element in the matrix and summing them up: $||W||_F^2 = \sum_{i} \sum_{j} (w_{ij})^2$. Essentially, it's the sum of the squares of all individual weight values in that layer.
• $\sum_{l=1}^{L}$ means we sum these penalties across all layers $l$ from $1$ to $L$.
• $m$ is the number of training examples in the mini-batch (or the total training set size, depending on the convention). Dividing by $m$ averages the penalty over the examples.
• $\lambda$ (lambda) is the regularization hyperparameter. This is a non-negative value that you choose before training. It controls the strength of the regularization. A larger $\lambda$ imposes a stronger penalty on large weights, leading to smaller weights and a simpler model. If $\lambda = 0$, we recover the original, unregularized loss function.
• The factor of $\frac{1}{2}$ is included primarily for mathematical convenience, as it simplifies the derivative calculation, which we'll see next.

The new, regularized loss function, let's call it $J_{reg}(\theta)$, becomes:

$$J_{reg}(\theta) = J(\theta) + \frac{\lambda}{2m} \sum_{l=1}^{L} ||W^{[l]}||_F^2$$

Note: Typically, bias terms (like $b^{[l]}$) are not included in the regularization penalty. While regularizing biases is possible, it often has a negligible effect on model complexity compared to regularizing weights and is commonly omitted in practice.

Impact on Gradient Descent

How does this additional term affect the training process via gradient descent? During backpropagation, we calculate the gradient of the loss function with respect to each parameter. With the L2 penalty added, the gradient of the regularized loss $J_{reg}$ with respect to a specific weight $w_{ij}^{[l]}$ (weight connecting neuron $j$ in layer $l-1$ to neuron $i$ in layer $l$) is:

$$\frac{\partial J_{reg}}{\partial w_{ij}^{[l]}} = \frac{\partial J(\theta)}{\partial w_{ij}^{[l]}} + \frac{\partial}{\partial w_{ij}^{[l]}} \left( \frac{\lambda}{2m} \sum_{k=1}^{L} ||W^{[k]}||_F^2 \right)$$

The derivative of the original loss term $\frac{\partial J(\theta)}{\partial w_{ij}^{[l]}}$ is calculated as usual via backpropagation. The derivative of the L2 penalty term with respect to $w_{ij}^{[l]}$ is:

$$\frac{\partial}{\partial w_{ij}^{[l]}} \left( \frac{\lambda}{2m} \sum_{k=1}^{L} \sum_{p} \sum_{q} (w_{pq}^{[k]})^2 \right) = \frac{\lambda}{2m} \frac{\partial}{\partial w_{ij}^{[l]}} (w_{ij}^{[l]})^2 = \frac{\lambda}{2m} (2 w_{ij}^{[l]}) = \frac{\lambda}{m} w_{ij}^{[l]}$$

(Note how the factor of 2 cancelled out conveniently).

So, the full gradient for a weight $w_{ij}^{[l]}$ is:

$$\frac{\partial J_{reg}}{\partial w_{ij}^{[l]}} = \frac{\partial J(\theta)}{\partial w_{ij}^{[l]}} + \frac{\lambda}{m} w_{ij}^{[l]}$$

The "Weight Decay" Effect

Now, let's look at the gradient descent update rule for this weight, using a learning rate $\alpha$:

$$w_{ij}^{[l]} := w_{ij}^{[l]} - \alpha \frac{\partial J_{reg}}{\partial w_{ij}^{[l]}}$$

Substituting the gradient we found:

$$w_{ij}^{[l]} := w_{ij}^{[l]} - \alpha \left( \frac{\partial J(\theta)}{\partial w_{ij}^{[l]}} + \frac{\lambda}{m} w_{ij}^{[l]} \right)$$

We can rearrange this slightly:

$$w_{ij}^{[l]} := w_{ij}^{[l]} - \alpha \frac{\lambda}{m} w_{ij}^{[l]} - \alpha \frac{\partial J(\theta)}{\partial w_{ij}^{[l]}}$$ $$w_{ij}^{[l]} := w_{ij}^{[l]} \left( 1 - \alpha \frac{\lambda}{m} \right) - \alpha \frac{\partial J(\theta)}{\partial w_{ij}^{[l]}}$$

Look closely at the term $\left( 1 - \alpha \frac{\lambda}{m} \right)$. Since the learning rate $\alpha$ and the regularization strength $\lambda$ are positive, and $m$ is the batch size (also positive), this factor is slightly less than 1 (assuming $\alpha \lambda / m$ is small enough, which it usually is for stable training).

This equation reveals why L2 regularization is often called weight decay. In each update step, before applying the gradient from the original loss, the weight $w_{ij}^{[l]}$ is multiplied by a factor slightly less than 1. This effectively shrinks or "decays" the weight towards zero at each step. The larger the weight, the larger the decay effect (due to the $\frac{\lambda}{m} w_{ij}^{[l]}$ term in the gradient). This mechanism discourages weights from growing too large, fulfilling the intuition we discussed earlier: keeping weights small helps to simplify the model and improve generalization.

The hyperparameter $\lambda$ directly controls the rate of this decay. A higher $\lambda$ leads to faster decay and smaller resulting weights. Finding the right value for $\lambda$ is a process of hyperparameter tuning, often done using a validation set.