Elastic Net: Combining L1 and L2

We've seen how L1 regularization encourages sparsity by driving some weights exactly to zero, effectively performing feature selection. We've also explored L2 regularization, often called weight decay, which prefers smaller weights overall and tends to shrink groups of correlated features together without necessarily making them zero.

What if you want a combination of these effects? Perhaps you suspect many features are irrelevant (suggesting L1), but you also believe there might be groups of correlated features that are important, and you don't want L1 to arbitrarily pick just one from the group (a scenario where L2 excels). This is where Elastic Net regularization comes in.

Elastic Net linearly combines the L1 and L2 penalties. Instead of just adding one type of penalty to the original loss function $L_{orig}(w)$, we add both:

$$L_{elastic}(w) = L_{orig}(w) + \lambda_1 ||w||_1 + \lambda_2 ||w||_2^2$$

Here, $||w||_1$ is the L1 norm (sum of absolute values of weights) and $||w||_2^2$ is the squared L2 norm (sum of squared weights). The hyperparameters $\lambda_1 \ge 0$ and $\lambda_2 \ge 0$ control the strength of the L1 and L2 penalties, respectively.

Often, you'll see this formulation parameterized differently in libraries, making it easier to tune the overall strength versus the mix:

$$L_{elastic}(w) = L_{orig}(w) + \alpha \left( \rho ||w||_1 + \frac{1 - \rho}{2} ||w||_2^2 \right)$$

In this common parameterization:

• $\alpha$ (alpha) is a non-negative hyperparameter controlling the overall strength of the regularization. If $\alpha = 0$, there's no regularization. Larger $\alpha$ increases the penalty.
• $\rho$ (rho, often called l1_ratio in libraries) is a hyperparameter between 0 and 1 that controls the mix between L1 and L2.
  • If $\rho = 1$, Elastic Net becomes pure L1 regularization.
  • If $\rho = 0$, Elastic Net becomes pure L2 regularization (scaled by $\alpha/2$).
  • If $0 < \rho < 1$, it's a mix of both.

Benefits of the Combination

Elastic Net aims to inherit the best of both worlds:

1. Sparsity: Like L1, it can perform feature selection by setting some weights to zero, especially useful when dealing with high-dimensional data where many features might be irrelevant.
2. Handles Correlated Features: Like L2, it tends to select groups of correlated features together. If features A and B are highly correlated and useful, L1 might arbitrarily zero out one of them, while Elastic Net (with $\rho < 1$) is more likely to shrink their weights similarly, possibly keeping both non-zero.
3. Stability: It often exhibits more stability than L1 regularization, particularly when the number of features ($p$) is much larger than the number of training samples ($n$), or when features are highly correlated.

Visualizing the Constraint

We can think of regularization as constraining the possible weight values. L1 regularization corresponds to a diamond-shaped constraint region (in 2D), while L2 corresponds to a circular region. Elastic Net creates a constraint region that blends these two shapes, having rounded corners but still favoring solutions along the axes compared to pure L2.

Comparison of the constraint regions for L1, L2, and Elastic Net regularization in two dimensions. Elastic Net combines the sharp corners of L1 (encouraging sparsity) with the roundness of L2.

Tuning Elastic Net

Using Elastic Net introduces two hyperparameters to tune: the overall strength $\alpha$ and the mixing ratio $\rho$ (l1_ratio). Finding the optimal combination usually requires techniques like grid search or random search over possible values for both parameters, using a validation set to evaluate performance. A common approach is to try a few discrete values for $\rho$ (e.g., 0.1, 0.5, 0.9, 1.0) and search for the best $\alpha$ for each $\rho$.

Implementation Considerations

While L2 regularization (weight decay) is often built directly into optimizers like Adam or SGD, implementing L1 or Elastic Net penalties in deep learning might require a bit more care depending on the framework.

• Layer-Specific Regularizers: Some frameworks (like Keras/TensorFlow) allow you to specify kernel_regularizer, bias_regularizer, etc., directly when defining a layer. You can often pass L1, L2, or Elastic Net regularizers here.
• Manual Addition to Loss: A more general approach, applicable in frameworks like PyTorch, is to calculate the L1 and L2 penalties for the relevant model parameters manually within the training loop and add them to the main computed loss before backpropagation.

Here's a conceptual PyTorch example of adding the penalty manually:

import torch
import torch.nn as nn

# Assume 'model' is your neural network
# Assume 'loss' is the computed loss from your criterion (e.g., CrossEntropyLoss)

alpha = 0.001 # Regularization strength
l1_ratio = 0.5 # Mixing parameter (rho)

l1_lambda = alpha * l1_ratio
l2_lambda = alpha * (1 - l1_ratio)

l1_penalty = 0.0
l2_penalty = 0.0

# Iterate over model parameters (weights)
for param in model.parameters():
    if param.requires_grad and param.dim() > 1: # Typically apply only to weight matrices
        l1_penalty += torch.linalg.norm(param, ord=1)
        l2_penalty += torch.linalg.norm(param, ord=2)**2 # L2 norm squared

# Combine original loss with penalties
total_loss = loss + l1_lambda * l1_penalty + (l2_lambda / 2) * l2_penalty
# (The 1/2 factor for L2 is common in the second formulation)

# Now proceed with backpropagation
# total_loss.backward()
# optimizer.step()

Note: This is a conceptual illustration. Efficient implementations often hook into the framework's gradient calculation or optimizer steps. Check your framework's documentation for recommended ways to apply combined L1/L2 penalties.

Elastic Net provides a flexible regularization option when you believe both sparsity and handling of correlated predictors are desirable. It requires tuning an additional hyperparameter but can lead to models that generalize better than those trained with only L1 or L2 regularization in certain situations.