Feature Scaling: Normalization and Standardization

Neural networks perform mathematical operations on input data. The scale and distribution of this data can significantly impact the network's ability to learn effectively. Imagine a dataset with two features: one ranging from 0 to 1, and another ranging from 1,000 to 100,000. During training, particularly when using gradient-based optimization methods, the feature with the larger values might disproportionately influence the updates to the network's weights, potentially slowing down convergence or even preventing the network from finding a good solution. Feature scaling addresses this by transforming the data so that all features contribute more equally to the learning process.

Two common techniques for feature scaling are Normalization (often called Min-Max Scaling) and Standardization (or Z-score Normalization). Let's examine each.

Normalization (Min-Max Scaling)

Normalization rescales features to a fixed range, typically [0, 1] or sometimes [-1, 1]. The formula for scaling to [0, 1] is:

x' = \frac{x - min(x)}{max(x) - min(x)}

Here, $x$ is the original feature value, $min(x)$ is the minimum value of that feature in the dataset, and $max(x)$ is the maximum value. Each value in the feature column is transformed into a new value $x'$ between 0 and 1.

Advantages:

Guarantees data falls within a specific range ([0, 1]). This can be useful for algorithms or activation functions (like Sigmoid) that expect inputs within this range.
Simple to understand and implement.

Disadvantages:

Highly sensitive to outliers. A single very large or very small value can significantly compress the rest of the data into a narrow band within the [0, 1] range, potentially reducing the usefulness of the scaling.
Doesn't handle varying standard deviations well if the goal is to make features comparable based on their spread.

Consider a feature with values ranging from 10 to 110. The minimum is 10, and the maximum is 110.

A value of 60 would be normalized to: $(60 - 10) / (110 - 10) = 50 / 100 = 0.5$ .
The minimum value 10 becomes: $(10 - 10) / (110 - 10) = 0 / 100 = 0$ .
The maximum value 110 becomes: $(110 - 10) / (110 - 10) = 100 / 100 = 1$ .

If an outlier of 1000 appeared, the maximum becomes 1000. Now, the value 60 normalizes to $(60 - 10) / (1000 - 10) = 50 / 990 \approx 0.05$ , compressing the original range significantly.

Standardization (Z-score Normalization)

Standardization rescales features so they have a mean ( $\mu$ ) of 0 and a standard deviation ( $\sigma$ ) of 1. The formula is:

x' = \frac{x - \mu}{\sigma}

Here, $x$ is the original value, $\mu$ is the mean of the feature values, and $\sigma$ is the standard deviation of the feature values. The resulting value $x'$ represents the number of standard deviations the original value $x$ was away from the mean.

Advantages:

Less affected by outliers compared to normalization because it uses the mean and standard deviation, which are inherently less sensitive to extreme values than min/max.
Centers the data around zero. This is often beneficial for optimization algorithms like gradient descent used in training neural networks.
Doesn't constrain data to a specific range, which might be preferable if outliers are meaningful and shouldn't be excessively compressed.

Disadvantages:

Doesn't produce values within a bounded interval, which might be a requirement for certain specific algorithms or situations (though generally not a problem for most neural network layers).

If a feature has a mean of 50 and a standard deviation of 15:

A value of 65 would be standardized to: $(65 - 50) / 15 = 15 / 15 = 1$ . (1 standard deviation above the mean)
A value of 35 would be standardized to: $(35 - 50) / 15 = -15 / 15 = -1$ . (1 standard deviation below the mean)
The mean value 50 becomes: $(50 - 50) / 15 = 0 / 15 = 0$ .

The following visualization shows the effect of Normalization and Standardization on a synthetic dataset with two features having different scales.

Scatter plot of original data with Feature 1 ranging roughly 20-80 and Feature 2 ranging roughly 1k-10k. Notice the different scales on the axes.

Effect of Normalization (top right) scaling data to [0, 1] on both axes, and Standardization (bottom left) centering data around (0, 0) with unit variance. The relative positions change slightly due to the outlier's influence.

Choosing Between Normalization and Standardization

There's no single definitive answer, but here are some guidelines:

Standardization is often preferred for neural network training. Techniques like gradient descent generally perform better when input features are centered around zero and have comparable variances. It's also more robust to outliers. Many standard network initializations and activation functions implicitly assume standardized inputs.
Normalization might be considered if:
- You need data strictly within a [0, 1] or [-1, 1] range, perhaps due to specific activation function choices (though modern activations like ReLU are less sensitive).
- The algorithm you are using specifically requires inputs in a bounded range.
- You are certain there are no significant outliers, or they have been handled separately.

In practice, standardization ( $x' = \frac{x - \mu}{\sigma}$ ) is the more common choice for preparing data for deep neural networks.

Important Implementation Note

A frequent mistake is to calculate scaling parameters (min/max or mean/std dev) across the entire dataset before splitting it into training, validation, and test sets. This introduces data leakage, where information from the validation and test sets influences the transformation applied to the training set, leading to overly optimistic performance estimates.

The correct procedure is:

Split your data into training, validation, and test sets.
Calculate the scaling parameters ( $min$ , $max$ , $\mu$ , $\sigma$ ) only from the training set.
Apply the same transformation (using the parameters derived from the training set) to the training, validation, and test sets.

This ensures that the model evaluation on the validation and test sets reflects how the model would perform on new, unseen data that has been processed using the knowledge gained solely from the training data. Libraries like scikit-learn provide tools (StandardScaler, MinMaxScaler) that facilitate this correct workflow using their fit (on training data) and transform (on all splits) methods.

Was this section helpful?

References

Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - This textbook provides a comprehensive overview of deep learning concepts, including the importance of data preprocessing techniques like feature scaling for network training.
The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Trevor Hastie, Robert Tibshirani, and Jerome Friedman, 2009 (Springer) - A classic reference for statistical learning, covering fundamental data preprocessing and scaling methods extensively.
Preprocessing data, scikit-learn developers, 2024 - Official documentation detailing various preprocessing techniques, including StandardScaler and MinMaxScaler, and best practices for their application to avoid data leakage.
Notes on Normalization, Andrew Ng, 2023 (DeepLearning.AI) - Concise notes from a leading deep learning educator, explaining data normalization techniques relevant to neural networks.