Interaction Features

Sometimes, the predictive ability of a feature isn't constant; it changes depending on the value of another feature. For instance, the impact of advertising spending on sales might be much stronger during holiday seasons than at other times of the year. Similarly, in a medical context, the effect of a medication might depend on a patient's age. Models, especially linear ones, might struggle to capture these combined effects using only the original features. This is where interaction features become valuable.

Interaction features are new features created by combining two or more existing features. The goal is to explicitly represent the combined effect or "interaction" between these variables. The most common way to create an interaction feature is by multiplying the values of the original features.

Capturing Combined Effects

Consider two features, $f_1$ and $f_2$. An interaction feature, $f_{interaction}$, can be created by multiplying them:

$f_{interaction} = f_1 \times f_2$

Why is this helpful? Let's look at a simple linear model:

$y = \beta_0 + \beta_1 f_1 + \beta_2 f_2$

In this model, the effect of $f_1$ on the target $y$ is always $\beta_1$, regardless of the value of $f_2$. Similarly, the effect of $f_2$ is always $\beta_2$.

Now, let's add the interaction term:

$y = \beta_0 + \beta_1 f_1 + \beta_2 f_2 + \beta_3 (f_1 \times f_2)$

The effect of a change in $f_1$ on $y$ is now $\beta_1 + \beta_3 f_2$. Notice that the effect of $f_1$ is no longer constant; it depends on the value of $f_2$. Likewise, the effect of $f_2$ on $y$ is $\beta_2 + \beta_3 f_1$, depending on the value of $f_1$. The interaction term allows the model to learn how the relationship between one feature and the target changes based on the level of another feature.

While multiplication is the most common operation for creating interactions, other operations like addition, subtraction, or division might sometimes be meaningful depending on the context, although they are used less frequently for standard interaction terms.

Implementation with Pandas

Creating basic interaction features in Pandas is straightforward using standard arithmetic operations on DataFrame columns.

import pandas as pd

# Sample DataFrame
data = {'feature_A': [1, 2, 3, 4, 5],
        'feature_B': [10, 8, 6, 4, 2],
        'target': [15, 21, 20, 19, 12]}
df = pd.DataFrame(data)

# Create an interaction feature by multiplying feature_A and feature_B
df['interaction_A_B'] = df['feature_A'] * df['feature_B']

print(df)
#    feature_A  feature_B  target  interaction_A_B
# 0          1         10      15               10
# 1          2          8      21               16
# 2          3          6      20               18
# 3          4          4      19               16
# 4          5          2      12               10

This new interaction_A_B column can now be used alongside the original features during model training.

Implementation with Scikit-learn

Scikit-learn's PolynomialFeatures transformer, primarily used for generating polynomial terms (which we'll cover next), can also be used to create interaction features specifically.

By setting the interaction_only parameter to True, PolynomialFeatures will generate only the products of combinations of features up to the specified degree. If degree=2, it will compute the product of all pairs of features. If degree=3, it will compute the product of all pairs and triplets of features.

import pandas as pd
from sklearn.preprocessing import PolynomialFeatures

# Sample DataFrame
data = {'feature_A': [1, 2, 3],
        'feature_B': [10, 8, 6],
        'feature_C': [0, 1, 0]}
df = pd.DataFrame(data)

# Initialize PolynomialFeatures to generate interaction terms only (degree 2)
# include_bias=False prevents adding a column of ones (intercept)
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)

# Fit and transform the data
interaction_features = poly.fit_transform(df[['feature_A', 'feature_B', 'feature_C']])

# Get the names of the new features
feature_names = poly.get_feature_names_out(['feature_A', 'feature_B', 'feature_C'])

# Create a DataFrame with the new features
df_interactions = pd.DataFrame(interaction_features, columns=feature_names)

print(df_interactions)
#    feature_A  feature_B  feature_C  feature_A feature_B  feature_A feature_C  feature_B feature_C
# 0        1.0       10.0        0.0             10.0                0.0                0.0
# 1        2.0        8.0        1.0             16.0                2.0                8.0
# 2        3.0        6.0        0.0             18.0                0.0                0.0

Notice that PolynomialFeatures outputs the original features along with the interaction terms (e.g., $f_A \times f_B$, $f_A \times f_C$, $f_B \times f_C$). If you only want the interaction terms themselves, you would need to select the appropriate columns after the transformation. If interaction_only is set to False (the default), PolynomialFeatures also generates polynomial terms like $f_A^2$, $f_B^2$, etc., in addition to the interaction terms.

Visualization Example

Imagine a classification problem where two classes (blue circles and red squares) are difficult to separate using only features $f_1$ and $f_2$ with a linear boundary.

Data points plotted based on their original features $f_1$ and $f_2$. A simple linear boundary struggles to separate the classes effectively.

Now, let's create an interaction feature $f_{interaction} = f_1 \times f_2$. If we plot $f_1$ against this new interaction feature, the classes might become linearly separable.

Data points plotted using $f_1$ and the interaction term $f_1 \times f_2$. The classes now appear more linearly separable.

This visualization illustrates how an interaction term can transform the feature space, potentially making it easier for certain models (especially linear ones) to find a good decision boundary.

Important Notes

While powerful, creating interaction features requires careful consideration:

1. Dimensionality: Generating interactions, especially between many features or using higher degrees (e.g., three-way interactions $f_1 \times f_2 \times f_3$), can dramatically increase the number of features in your dataset. This can slow down model training and increase the risk of overfitting (the curse of dimensionality).
2. Multicollinearity: Interaction terms are inherently correlated with the original features they were derived from. High multicollinearity can sometimes make model coefficients unstable and difficult to interpret, although techniques like regularization can mitigate this.
3. Overfitting: Adding many interaction features gives the model more flexibility, potentially allowing it to fit the noise in the training data too closely, leading to poor performance on unseen data. Feature selection or regularization techniques (like Lasso L1) become important to select only the useful interactions or penalize complex models.
4. Interpretability: While interaction terms allow models to capture complex relationships, interpreting the magnitude and sign of the coefficient associated with an interaction term ($\beta_3$ in our earlier example) requires careful thought about how the combined effect influences the target.
5. Feature Scaling: If the original features have significantly different scales, their product might be dominated by the feature with the larger magnitude. It's often advisable to scale features (e.g., using StandardScaler or MinMaxScaler) before creating multiplicative interaction terms.

When to Use Them

Consider creating interaction features when:

• You have domain knowledge suggesting that the combined effect of certain features is important.
• Your current model (especially a linear model) is underperforming, and you suspect non-linear relationships involving combinations of features exist.
• You are prepared to manage the increased dimensionality through techniques like regularization or feature selection.

Creating interaction features is often an iterative process. You might hypothesize certain interactions based on data exploration or domain knowledge, build models with them, evaluate performance, and refine your feature set accordingly. They represent a way to manually inject non-linearity and combined effects into your feature set, potentially leading to more accurate and insightful models.