L1 Regularization: Mathematical Formulation

L1 regularization modifies the standard loss function by adding a penalty term proportional to the sum of the absolute values of all the weights in the network.

Recall that our goal during training is typically to minimize a data loss function, let's call it $L_{data}(W)$, which measures how well the model's predictions match the actual target values given the current weights $W$. With L1 regularization, we add a new term to this objective:

$$L_{total}(W) = L_{data}(W) + \lambda \sum_{i} |w_i|$$

Let's break down this new term:

• $w_i$: Represents an individual weight within the network (across all layers).
• $\sum_{i} |w_i|$: This is the sum of the absolute values of all weights. It's also known as the $L_1$ norm of the weight vector, often written as $||W||_1$.
• $\lambda$: This is the regularization strength hyperparameter, identical in role to the $\lambda$ used in L2 regularization. It's a non-negative value (typically small) that controls the influence of the L1 penalty. A higher $\lambda$ imposes a stronger penalty, encouraging more sparsity. If $\lambda = 0$, we recover the original unregularized loss function.

The core difference compared to L2 regularization lies in the use of the absolute value $|w_i|$ instead of the squared value $w_i^2$. This seemingly small change has a significant impact on the optimization process.

Impact on Gradient Descent

During backpropagation, we need to compute the gradient of the total loss $L_{total}$ with respect to each weight $w_j$ to update it. The gradient of the L1 penalty term with respect to a specific weight $w_j$ is:

$$\frac{\partial}{\partial w_j} \left( \lambda \sum_{i} |w_i| \right) = \lambda \cdot \frac{\partial |w_j|}{\partial w_j}$$

The derivative of the absolute value function $|x|$ is $\text{sign}(x)$, which is:

• $+1$ if $x > 0$
• $-1$ if $x < 0$
• Undefined at $x = 0$

So, the gradient of the total loss becomes:

$$\frac{\partial L_{total}}{\partial w_j} = \frac{\partial L_{data}}{\partial w_j} + \lambda \cdot \text{sign}(w_j) \quad \text{for } w_j \neq 0$$

This means the L1 penalty adds a constant value ($\lambda$ or $-\lambda$) to the gradient of the data loss, pushing the weight towards zero regardless of the weight's current magnitude (as long as it's not zero). This constant push is what drives weights to become exactly zero. Compare this to L2 regularization, where the penalty's contribution to the gradient ($\lambda \cdot 2w_j$) decreases as the weight gets smaller, making it less likely to reach precisely zero.

Handling Non-Differentiability at Zero

What happens when $w_j = 0$? The absolute value function isn't differentiable at this point. In practice, optimization algorithms handle this using techniques related to subgradient descent or proximal gradient methods. The subgradient of $|w_j|$ at $w_j=0$ is the interval $[-1, 1]$. A common practical approach is to simply set the gradient contribution from the L1 term to zero if the weight is already zero, or to apply a "soft thresholding" operation during the update step, which effectively checks if the update would cross zero and sets the weight to zero if it does. Most deep learning framework optimizers handle this detail internally.

The essential takeaway is that the L1 penalty provides a constant "push" towards zero for non-zero weights, making it highly effective at producing sparse models where many weights become exactly zero.

The L1 penalty forms a 'V' shape, applying a constant gradient magnitude (slope) regardless of the weight's size when it's non-zero. The L2 penalty forms a parabola, with a gradient that decreases as the weight approaches zero.

This mathematical structure directly leads to the sparsity-inducing property discussed earlier. In the next sections, we'll compare L1 and L2 more directly and look at how to implement them in code.