Embedded Methods: Regularization (Lasso L1)

Embedded methods integrate the feature selection process directly within the model training phase. Unlike filter methods that assess features independently or wrapper methods that repeatedly train a model on different subsets, embedded techniques perform feature selection as part of learning the model parameters. This approach can be computationally more efficient than wrapper methods and often captures feature interactions better than filter methods.

One of the most prominent examples of embedded feature selection is L1 regularization, commonly known as Lasso (Least Absolute Shrinkage and Selection Operator). Lasso is typically applied to linear models (like Linear Regression or Logistic Regression).

Understanding Regularization

Before looking specifically at Lasso, let's briefly touch upon regularization. In machine learning, especially with linear models, we aim to find model parameters (coefficients) that minimize a loss function (like Mean Squared Error for regression). However, simply minimizing the training error can lead to complex models that overfit the training data and perform poorly on unseen data.

Regularization adds a penalty term to the loss function. This penalty discourages overly complex models, typically by constraining the magnitude of the coefficients. The model now tries to minimize:

$$\text{Loss Function} + \text{Penalty Term}$$

The Lasso (L1) Penalty

Lasso uses the $L1$ norm of the coefficient vector as its penalty term. For a linear model with coefficients $\beta = (\beta_1, \beta_2, ..., \beta_p)$ for $p$ features, the $L1$ norm is the sum of the absolute values of the coefficients:

$$||\beta||_1 = \sum_{j=1}^{p} |\beta_j|$$

The objective function for Lasso Regression (minimizing Mean Squared Error with an L1 penalty) becomes:

$$\underset{\beta}{\operatorname{argmin}} \left( \sum_{i=1}^{n} (y_i - \sum_{j=1}^{p} x_{ij}\beta_j)^2 + \alpha \sum_{j=1}^{p} |\beta_j| \right)$$

Here:

• $n$ is the number of samples.
• $p$ is the number of features.
• $y_i$ is the target value for the $i$-th sample.
• $x_{ij}$ is the value of the $j$-th feature for the $i$-th sample.
• $\beta_j$ is the coefficient for the $j$-th feature.
• $\alpha$ (alpha) is the regularization strength parameter. It controls the trade-off between fitting the data well (minimizing the sum of squared errors) and keeping the coefficients small (minimizing the L1 penalty).

How Lasso Performs Feature Selection

The main aspect of the $L1$ penalty is its tendency to shrink some coefficients to exactly zero. This happens because of the shape of the L1 constraint (a diamond in two dimensions, a multi-dimensional equivalent in higher dimensions) when optimizing the loss function. Features whose coefficients are shrunk to zero are effectively removed from the model.

• Higher $\alpha$: Increases the penalty on large coefficients. More coefficients are likely to be shrunk to zero, resulting in a simpler model with fewer features selected.
• Lower $\alpha$: Reduces the penalty. Coefficients are shrunk less, and fewer (or none) might become exactly zero. If $\alpha = 0$, Lasso reduces to standard Linear Regression (without regularization).

This property makes Lasso an embedded feature selection technique: the process of fitting the model itself selects the relevant features by assigning non-zero coefficients only to them.

Illustration of how Lasso coefficients change as the regularization strength (alpha) increases. Notice how some coefficients (Feature 2, Feature 3, Feature 4) are shrunk to exactly zero at different alpha values, effectively removing them from the model. Feature 1 persists longer but also shrinks.

Implementing Lasso in Scikit-learn

Scikit-learn provides the Lasso class for regression tasks. You can fit it like any other Scikit-learn model and then inspect the coef_ attribute to see which features were assigned non-zero coefficients.

import numpy as np
from sklearn.linear_model import Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_regression

# Generate synthetic data
X, y = make_regression(n_samples=200, n_features=20, n_informative=10, noise=15, random_state=42)

# Introduce some irrelevant features (making last 5 features redundant)
X[:,-5:] = X[:,:5] * np.random.uniform(0.9, 1.1, size=(X.shape[0], 5))

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features (Important for Lasso!)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Initialize and fit Lasso model
# Adjust alpha to control sparsity (higher alpha = more zeros)
alpha_value = 1.0
lasso = Lasso(alpha=alpha_value, random_state=42)
lasso.fit(X_train_scaled, y_train)

# Inspect coefficients
print(f"Lasso Coefficients (alpha={alpha_value}):")
print(lasso.coef_)

# Identify selected features (non-zero coefficients)
selected_features_mask = lasso.coef_ != 0
n_selected_features = np.sum(selected_features_mask)
print(f"\nNumber of features selected: {n_selected_features} out of {X_train_scaled.shape[1]}")

# Get indices of selected features
selected_indices = np.where(selected_features_mask)[0]
print(f"Indices of selected features: {selected_indices}")

# You can now use these selected features for subsequent modeling
# X_train_selected = X_train_scaled[:, selected_features_mask]
# X_test_selected = X_test_scaled[:, selected_features_mask]

Important Note: Lasso, like many regularized models and distance-based algorithms, is sensitive to the scale of the input features. It's standard practice to scale your data (e.g., using StandardScaler or MinMaxScaler) before applying Lasso.

Using Lasso for General Feature Selection

You don't have to use Lasso solely for building the final prediction model. It can be used as a feature selection step within a larger pipeline. Scikit-learn's SelectFromModel meta-transformer makes this easy. It selects features based on importance weights (like coefficients from Lasso or feature importances from trees).

from sklearn.feature_selection import SelectFromModel
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression # Example for classification

# --- Assuming X_train_scaled, y_train are ready ---
# Example for classification - use LogisticRegression with L1 penalty

# Create a pipeline: Lasso (as selector) -> Final Model
# Use LassoCV or LogisticRegressionCV to find best alpha internally, or set alpha manually
lasso_selector_model = Lasso(alpha=1.0, random_state=42) # Or LogisticRegression(penalty='l1', C=1.0, solver='liblinear') for classification

selector = SelectFromModel(estimator=lasso_selector_model, threshold="median") # Select features with coef > median coef

# Example pipeline with Logistic Regression as the final estimator
# For classification, you might use LogisticRegression(penalty='l1', C=0.1, solver='liblinear') directly in SelectFromModel
# For regression, you could follow with another regressor like Ridge or LinearRegression
pipeline = Pipeline([
  ('scaler', StandardScaler()), # Ensure scaling happens within the pipeline
  ('feature_selection', selector),
  ('classification', LogisticRegression(random_state=42)) # Example final model
])

# Fit the pipeline (selector and classifier trained)
# pipeline.fit(X_train, y_train) # Assuming y_train is categorical for LogisticRegression

# Now 'pipeline' represents the full process including L1 feature selection

Pros and Cons of Lasso (L1) Feature Selection

Pros:

• Automatic Feature Selection: Built directly into the model fitting process.
• Handles Collinearity: If multiple features are highly correlated, Lasso tends to arbitrarily pick one and shrink the others to zero, which can be beneficial.
• Produces Sparse Models: The resulting models are often simpler and easier to interpret because many coefficients are zero.
• Computationally Efficient: Generally faster than wrapper methods, especially for high-dimensional data.

Cons:

• Sensitivity to $\alpha$: The number of selected features heavily depends on the choice of the regularization parameter $\alpha$. Cross-validation (e.g., using LassoCV) is typically needed to find a good value.
• May Struggle with Groups of Correlated Features: While it handles collinearity by picking one feature, it might not be the "best" one, and sometimes you might prefer to keep correlated features if they represent a meaningful group. Elastic Net regularization (a combination of L1 and L2) can sometimes handle this better.
• Performance: If most features are actually relevant to the target variable, Lasso's tendency to shrink coefficients might hurt predictive performance compared to methods like Ridge (L2) regularization, which shrinks coefficients towards zero but rarely makes them exactly zero.

Lasso provides a powerful and widely used embedded method for feature selection, particularly effective when dealing with high-dimensional datasets where many features might be irrelevant. Remember to scale your data and carefully tune the alpha parameter for optimal results.