Selecting Features using Statistical Methods

Datasets frequently contain a large number of input variables. This often occurs after features are engineered from raw numerical, categorical, and text data, or when transformations such as PCA are applied. Having more information might seem beneficial. However, not all features are created equal. Some might be irrelevant to the prediction task, others might be redundant (highly correlated with existing features), and some might even introduce noise that degrades model performance. Furthermore, using too many features can increase computational cost and model complexity, sometimes leading to overfitting (the "curse of dimensionality").

This is where feature selection comes into play. Instead of transforming features into a lower-dimensional space like PCA does, feature selection techniques aim to identify and retain a subset of the original, most relevant features for your modeling task. Statistical methods provide a common and often computationally efficient approach to performing this selection, primarily by evaluating the relationship between each feature and the target variable independently of any specific machine learning model. These are often referred to as filter methods.

Filter Methods for Feature Selection

Filter methods assess the relevance of features based on their intrinsic statistical properties concerning the target variable. They are generally faster than other selection strategies (like wrapper or embedded methods) because they don't involve training machine learning models during the selection process. They serve as an effective preprocessing step to filter out features that are unlikely to be useful.

The choice of statistical test largely depends on the data types of the feature and the target variable.

Evaluating Features for Regression Tasks (Continuous Target)

When your target variable is continuous, you typically want to select features that show a strong statistical relationship with it.

1. Pearson Correlation Coefficient: This measures the linear correlation between a numerical feature ($X$) and the continuous target variable ($Y$). The correlation coefficient, $r$, ranges from -1 to +1.

   • $r \approx +1$: Strong positive linear relationship.
   • $r \approx -1$: Strong negative linear relationship.
   • $r \approx 0$: Weak or no linear relationship. Features with higher absolute correlation values are often considered more relevant. You can easily compute this using Pandas: df.corr()['target_variable'].abs().sort_values(ascending=False). However, remember that Pearson correlation only captures linear dependencies. A feature might have a strong non-linear relationship with the target but a low Pearson correlation score.

2. ANOVA F-test (Regression): This statistical test (sklearn.feature_selection.f_regression) computes the F-statistic to determine if there's a significant linear relationship between a numerical feature and the continuous target variable. It essentially analyzes the variance explained by the feature. A higher F-statistic (and correspondingly lower p-value) suggests a stronger relationship. It's fundamentally linked to linear regression and Pearson correlation for single features.

3. Mutual Information (Regression): Mutual information (sklearn.feature_selection.mutual_info_regression) is a non-parametric method derived from information theory. It measures the amount of information obtained about one variable (the target) by observing another variable (the feature). It can capture arbitrary relationships (not just linear) and is measured in "nats" (or sometimes bits). A score of 0 indicates independence, while higher values indicate greater dependence. This makes it powerful for capturing complex patterns that correlation might miss. It requires numerical features, but categorical features can sometimes be used if appropriately encoded or if the implementation supports it (often requiring discretization for continuous variables).

Evaluating Features for Classification Tasks (Categorical Target)

When dealing with a categorical target variable, different statistical tests are appropriate.

1. Chi-Squared ($\chi^2$) Test: This test (sklearn.feature_selection.chi2) is used to determine if there is a significant association between two categorical variables. You would apply this between each categorical feature and the categorical target. It compares the observed frequencies in a contingency table (feature categories vs. target classes) to the frequencies that would be expected if the variables were independent. A high $\chi^2$ statistic suggests that the feature is not independent of the target class, making it potentially relevant. An important prerequisite is that the features must contain non-negative values (counts, frequencies, or appropriately encoded variables like one-hot encoded features).

2. ANOVA F-test (Classification): Similar to the regression case, but adapted for classification (sklearn.feature_selection.f_classif). It tests whether the mean values of a numerical feature differ significantly across the different target classes. If a feature's mean value varies substantially between classes, it's likely a good predictor for distinguishing those classes.

3. Mutual Information (Classification): Analogous to the regression version, sklearn.feature_selection.mutual_info_classif measures the mutual dependence between each feature (can be numerical or discrete) and the categorical target variable. It quantifies how much uncertainty about the target class is reduced by knowing the value of the feature. Higher values indicate greater relevance, and it effectively captures non-linear relationships.

Implementing Statistical Feature Selection with Scikit-learn

Scikit-learn's sklearn.feature_selection module provides convenient tools for applying these statistical tests. Common selectors include SelectKBest (selects the top $k$ features based on scores) and SelectPercentile (selects the top features based on a percentile threshold).

Here's how you might use SelectKBest with the ANOVA F-test for a classification problem:

import pandas as pd
from sklearn.datasets import make_classification
from sklearn.feature_selection import SelectKBest, f_classif

# Generate synthetic classification data
# 10 informative features, 10 redundant/noisy features
X, y = make_classification(n_samples=200, n_features=20, n_informative=10,
                           n_redundant=5, n_classes=3, random_state=42)
X_df = pd.DataFrame(X, columns=[f'feature_{i}' for i in range(20)])

print("Original number of features:", X_df.shape[1])

# Select the top 12 features based on ANOVA F-test scores
k_best = 12
selector = SelectKBest(score_func=f_classif, k=k_best)

# Fit the selector to the data and transform X
X_new = selector.fit_transform(X_df, y)

# Get the scores and p-values
scores = selector.scores_
p_values = selector.pvalues_

# Get the indices of the selected features
selected_indices = selector.get_support(indices=True)

# Get the names of the selected features
selected_features = X_df.columns[selected_indices]

print(f"Selected top {k_best} features:", selected_features.tolist())
print("Shape of data after selection:", X_new.shape)

# Optional: Display scores for selected features
# selected_scores = pd.Series(scores[selected_indices], index=selected_features)
# print("\nScores (F-statistic) for selected features:")
# print(selected_scores.sort_values(ascending=False))

This code snippet demonstrates:

1. Generating sample data.
2. Initializing SelectKBest with f_classif as the scoring function and specifying $k=12$.
3. Fitting the selector to the data (X_df, y).
4. Transforming the data to keep only the selected features (X_new).
5. Retrieving the names of the features that were selected.

You could similarly use chi2 (for non-negative categorical features), mutual_info_classif, f_regression, or mutual_info_regression as the score_func depending on your feature types and task (classification/regression).

Example F-statistic scores calculated by SelectKBest using f_classif. Features with higher scores exhibit stronger statistical relationships with the target classes in this example classification scenario. The selector retains the features corresponding to the top $k$ scores.

Filter Methods

While powerful, keep these points in mind when using statistical filter methods:

• Independence Assumption: Most filter methods evaluate each feature independently of others. They don't account for interactions or redundancy between features. A feature might score low individually but be valuable when combined with another feature. Conversely, multiple highly correlated features might all get high scores, leading to redundancy in the selected set. It's often useful to analyze feature correlations (X_df[selected_features].corr()) after selection.
• Choice of $k$: Determining the optimal number of features ($k$) is not always straightforward. You might choose $k$ based on a desired dimensionality, domain knowledge, or more systematically by evaluating model performance with different values of $k$ using cross-validation (though this starts blending into wrapper method ideas). Using SelectPercentile offers an alternative way to specify the number of features.
• Test Assumptions: Be mindful of the underlying assumptions of the statistical test (e.g., linearity for Pearson correlation, non-negativity for $\chi^2$).
• Scaling: While correlation and mutual information are less sensitive, variance-based methods like ANOVA F-tests can be affected by the scale of numerical features. Applying standard scaling (StandardScaler) before feature selection is often a good practice for these methods.
• Model Dependence: Filter methods are model-agnostic. The feature set selected might not be the absolute best for a specific machine learning algorithm (e.g., tree-based models might handle irrelevant features differently than linear models). Wrapper and embedded methods address this but are computationally more expensive.

Statistical feature selection provides a valuable set of techniques for reducing the dimensionality of your dataset by focusing on features with the strongest individual relationships to the target variable. It's an important step in the practical data science workflow, helping to simplify models, potentially improve performance, reduce computation time, and prepare a more focused dataset for the subsequent modeling stages.