Generating Features from Numerical Data

Raw numerical features, while seemingly straightforward, often don't reveal their full potential to machine learning algorithms directly. Models like linear regression assume linear relationships, while others might struggle with skewed distributions or large ranges. Generating new features from existing numerical ones can help models better capture underlying patterns, handle non-linearities, and improve overall performance.

Binning (Discretization)

Sometimes, the exact numerical value of a feature isn't as important as the range or bin it falls into. Binning, or discretization, involves converting continuous numerical features into discrete categorical ones by grouping values into predefined bins. This can be useful for several reasons:

1. Capturing Non-linear Effects: Linear models can only capture linear relationships. By binning a feature, you allow the model to assign different weights or importance to different ranges, effectively capturing non-linear patterns.
2. Reducing Sensitivity to Outliers: Extreme values can disproportionately influence some models. Binning places outliers into specific bins (often the first or last), reducing their direct impact.
3. Simplifying Models: For some models like Decision Trees, binning can simplify the structure by providing natural split points.

There are two main approaches to binning:

• Fixed-Width Binning: Divides the range of the data into bins of equal width. The number of data points in each bin can vary significantly.
• Quantile-Based Binning: Divides the data so that each bin contains approximately the same number of data points. The bin widths will vary based on the data distribution.

Let's see how to implement this using Pandas. Assume we have a DataFrame df with an Age column.

import pandas as pd
import numpy as np

# Sample data
data = {'Age': [22, 25, 31, 45, 58, 62, 75, 81, 19, 38]}
df = pd.DataFrame(data)

# 1. Fixed-Width Binning (e.g., 4 bins)
# Define bin edges explicitly
bins = [18, 30, 45, 60, 100] # Ages 18-30, 31-45, 46-60, 61-100
labels = ['18-30', '31-45', '46-60', '61+']
df['Age_Bin_Fixed'] = pd.cut(df['Age'], bins=bins, labels=labels, right=True, include_lowest=True)

# Or let pandas determine equal-width bins
# df['Age_Bin_Fixed_Auto'] = pd.cut(df['Age'], bins=4) # Creates 4 equal-width bins

# 2. Quantile-Based Binning (e.g., 4 quantiles/quartiles)
df['Age_Bin_Quantile'] = pd.qcut(df['Age'], q=4, labels=['Q1', 'Q2', 'Q3', 'Q4'])

print(df)

Choosing the Number of Bins: The number of bins is a hyperparameter. Too few bins might oversimplify the data, losing valuable information. Too many bins might make the feature too granular, approaching the original continuous variable and potentially leading to overfitting. Cross-validation can help determine an appropriate number of bins.

Polynomial Features

"Linear models, by definition, model linear relationships ($y = w_1 x_1 + w_2 x_2 + ... + b$). However, relationships are often non-linear. Polynomial features create new features by raising existing features to a power (e.g., $x^2$, $x^3$) and creating interaction terms between features (e.g., $x_1 \cdot x_2$)."

Consider a simple feature $x$. Adding $x^2$ allows a linear model to fit a quadratic relationship: $y = w_1 x + w_2 x^2 + b$. This is still a linear model because it's linear with respect to the coefficients ($w_1, w_2, b$), even though the relationship between $y$ and the original $x$ is non-linear.

Scikit-learn's PolynomialFeatures transformer is a convenient way to generate these.

from sklearn.preprocessing import PolynomialFeatures
import pandas as pd

# Sample data with two features
data = {'FeatureA': [1, 2, 3, 4, 5],
        'FeatureB': [2, 3, 5, 5, 7]}
df_poly = pd.DataFrame(data)

# Create polynomial features up to degree 2
# include_bias=False avoids adding a column of ones (intercept)
poly = PolynomialFeatures(degree=2, include_bias=False)
poly_features = poly.fit_transform(df_poly)

# Get feature names for clarity
feature_names = poly.get_feature_names_out(df_poly.columns)

# Create a new DataFrame with these features
df_poly_transformed = pd.DataFrame(poly_features, columns=feature_names)

print("Original Features:")
print(df_poly)
print("\nPolynomial Features (degree=2):")
print(df_poly_transformed)

# Example for degree=3, only FeatureA
poly_deg3 = PolynomialFeatures(degree=3, include_bias=False)
poly_features_a = poly_deg3.fit_transform(df_poly[['FeatureA']])
feature_names_a = poly_deg3.get_feature_names_out(['FeatureA'])
df_poly_a_transformed = pd.DataFrame(poly_features_a, columns=feature_names_a)

print("\nPolynomial Features (degree=3, FeatureA only):")
print(df_poly_a_transformed)

Considerations:

• Dimensionality: The number of generated features grows rapidly with the degree and the number of original features. A degree $d$ polynomial for $n$ features creates $\binom{n+d}{d} - 1$ features (excluding the bias term). This can lead to high computational cost and overfitting (the curse of dimensionality).
• Feature Scaling: It's often essential to scale features before applying polynomial transformations, especially for higher degrees, to avoid numerical instability and issues with regularization.
• Interpretability: While polynomial features can improve model performance, they make the resulting model harder to interpret, as the coefficients now relate to transformed features ($x^2$, $x_1 \cdot x_2$, etc.).

Mathematical Transformations

Applying mathematical functions like logarithms, square roots, or reciprocals can help stabilize variance, make distributions more symmetric (closer to normal), and handle skewed data. Many machine learning algorithms perform better when features follow a distribution closer to a Gaussian (normal) distribution.

• Log Transformation ($log(x)$): Useful for highly skewed data, especially right-skewed data (long tail to the right). It compresses the range of large values and expands the range of small values. Requires data to be strictly positive ($x > 0$). If you have zeros or negative numbers, you might use $log(x+c)$ where $c$ is a small constant (like 1) to make all values positive.
• Square Root Transformation ($\sqrt{x}$): Also helps with right-skewed data, but the effect is weaker than the log transform. Applicable for non-negative data ($x \ge 0$).
• Box-Cox Transformation: A family of power transformations that includes log and square root as special cases. It finds an optimal parameter ($\lambda$) to transform the data to be as close to normally distributed as possible. Requires strictly positive data ($x > 0$). $$y^{(\lambda)} = \begin{cases} \frac{x^\lambda - 1}{\lambda} & \text{if } \lambda \neq 0 \\ \log(x) & \text{if } \lambda = 0 \end{cases}$$
• Reciprocal Transformation ($1/x$): Can be useful in specific cases but is less common. Also requires $x \neq 0$.

Let's apply log and square root transformations using NumPy.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats

# Sample skewed data (e.g., income)
np.random.seed(42)
income_data = np.random.gamma(2, scale=2000, size=1000)
income_data = np.round(income_data, 2)
df_trans = pd.DataFrame({'Income': income_data})

# Add a small constant for log transform if zeros are possible (not needed here)
# df_trans['Income_Log'] = np.log(df_trans['Income'] + 1)

# Apply Log Transformation
df_trans['Income_Log'] = np.log(df_trans['Income'])

# Apply Square Root Transformation
df_trans['Income_Sqrt'] = np.sqrt(df_trans['Income'])

# Apply Box-Cox Transformation
# scipy.stats.boxcox returns the transformed data and the optimal lambda
df_trans['Income_BoxCox'], best_lambda = stats.boxcox(df_trans['Income'])

print(f"Optimal lambda for Box-Cox: {best_lambda:.4f}")
print(df_trans[['Income', 'Income_Log', 'Income_Sqrt', 'Income_BoxCox']].head())

# --- Visualization (Optional: Use Matplotlib or Plotly for distribution plots) ---
# Example using Matplotlib (Plotly JSON can be large for distributions)
fig, axes = plt.subplots(1, 4, figsize=(16, 4))
axes[0].hist(df_trans['Income'], bins=30, color='#4dabf7')
axes[0].set_title('Original Income')
axes[1].hist(df_trans['Income_Log'], bins=30, color='#9775fa')
axes[1].set_title('Log Transformed')
axes[2].hist(df_trans['Income_Sqrt'], bins=30, color='#69db7c')
axes[2].set_title('Square Root Transformed')
axes[3].hist(df_trans['Income_BoxCox'], bins=30, color='#ff922b')
axes[3].set_title('Box-Cox Transformed')
plt.tight_layout()
# If using in a web context, you'd save this plot or generate a Plotly version.
# plt.show() # Uncomment to display plot locally

The histograms illustrate how transformations can reduce skewness. The original income data is heavily right-skewed. Log and Box-Cox transformations produce distributions much closer to symmetrical, while the square root transformation offers a milder effect.

When to Apply Transformations:

• Observe feature distributions using histograms or Q-Q plots.
• Check if model residuals show patterns related to feature magnitudes (indicative of heteroscedasticity).
• Some models (like linear regression, logistic regression, SVMs with certain kernels, K-Means) often benefit from features that are closer to normally distributed or have similar scales. Tree-based models are generally less sensitive to feature transformations and scaling.

Generating features from numerical data is often an iterative process. You might try binning, polynomial features, or transformations, train a model, evaluate its performance, and then refine your feature engineering strategy based on the results. The techniques discussed here provide a solid foundation for creating more informative numerical features for your machine learning models.