Splitting Data into Training and Testing Sets

Splitting data into training and testing sets is a critical step before training any machine learning model. This process typically follows data preparation, which often includes cleaning and engineering features. Why is data splitting so important? The primary goal of building a machine learning model is to have it perform well on new, unseen data. If the model is trained and evaluated on the exact same data, there is no realistic way of knowing how it will generalize to data it has not encountered before.

The Problem: Evaluating Generalization

Imagine studying for an exam using only the exact questions and answers that will be on the final test. You might memorize the answers perfectly and achieve a 100% score, but this doesn't mean you've actually learned the underlying concepts or could answer different questions on the same topic. Similarly, a machine learning model can "memorize" the training data, including its noise and specific patterns. This phenomenon is called overfitting. An overfit model performs exceptionally well on the data it was trained on but fails miserably when presented with new data.

To get an honest assessment of how our model is likely to perform in reality, we need a way to simulate encountering unseen data. This is achieved by partitioning our dataset into at least two distinct subsets:

Training Set: This subset of the data is used to actually train the model. The model learns patterns, relationships, and adjusts its internal parameters based on this data.
Testing Set (or Hold-out Set): This subset is kept separate throughout the training process. It acts as the "unseen" data to evaluate the final model's performance after training is complete. Evaluating on the test set gives us an estimate of the model's generalization ability.

Implementing the Split with Scikit-learn

The most common way to perform this split in Python is using the train_test_split function from Scikit-learn's model_selection module. It's a versatile function that handles shuffling and splitting arrays or matrices efficiently.

Let's assume you have your features (the input variables for the model) in a Pandas DataFrame or NumPy array called X, and your target variable (what you want to predict) in a Pandas Series or NumPy array called y.

import pandas as pd
from sklearn.model_selection import train_test_split

# Assume 'X' contains your features and 'y' contains the target variable
# Example:
# X = dataframe[['feature1', 'feature2', 'feature3']]
# y = dataframe['target_label']

# Split the data into training and testing sets
# test_size=0.2 means 20% of the data will be used for the test set
# random_state ensures reproducibility - using the same number will always
# produce the same split.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(f"Shape of X_train: {X_train.shape}")
print(f"Shape of X_test: {X_test.shape}")
print(f"Shape of y_train: {y_train.shape}")
print(f"Shape of y_test: {y_test.shape}")

Parameters of train_test_split:

*arrays: The sequences of arrays (features X, target y) to be split. They must all have the same length along the first axis (same number of samples).
test_size: Specifies the proportion of the dataset to include in the test split. It can be a float between 0.0 and 1.0 (representing the proportion) or an int (representing the absolute number of test samples). If train_size is also specified, test_size is often inferred. Common values are 0.2, 0.25, or 0.3.
train_size: Similar to test_size, but specifies the proportion or number for the training set. If None, it's set to the complement of test_size.
random_state: Controls the shuffling applied to the data before applying the split. Passing an integer ensures that the split is the same every time the code is run, which is essential for reproducibility. If None, the split will be different each time.
shuffle: A boolean indicating whether to shuffle the data before splitting. This is True by default and is generally recommended unless you have a specific reason not to (like time-series data where order matters). Shuffling helps ensure that the training and testing sets are representative of the overall data distribution, especially if the original data is sorted or ordered in some way.
stratify: This is particularly important for classification problems. If your target variable y represents classes, setting stratify=y ensures that the proportion of values for each class is approximately the same in both the training and testing sets as it is in the original dataset. This is important when dealing with imbalanced datasets where one class might be much rarer than others. Without stratification, a random split could potentially result in a test set having very few or even zero instances of a rare class.

Stratified Splitting Example

Imagine a classification task where 80% of your data belongs to Class A and 20% to Class B. A simple random split might accidentally put, say, 90% of Class B instances in the test set. This would make the training set less representative and the test set evaluation potentially misleading. Stratification prevents this.

# Example assuming 'y' contains class labels for a classification problem
X_train_strat, X_test_strat, y_train_strat, y_test_strat = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42,
    stratify=y  # Ensure class proportions are maintained
)

# You can verify the proportions (e.g., using pandas value_counts())
print("Original Class Distribution:")
print(y.value_counts(normalize=True))

print("\nTraining Set Class Distribution:")
print(y_train_strat.value_counts(normalize=True))

print("\nTest Set Class Distribution:")
print(y_test_strat.value_counts(normalize=True))

When to Split: Avoiding Data Leakage

A common issue is applying data transformations (like scaling features using StandardScaler or imputing missing values using the mean) before splitting the data. If you calculate the mean or standard deviation using the entire dataset and then use these values to scale both the training and testing sets, information from the test set implicitly "leaks" into the training process. The model effectively gets a sneak peek at the test data's distribution through these calculated parameters.

The correct procedure is generally:

Split the data into training and testing sets first.
Fit any transformers (like scalers, imputers, encoders) only on the training data (X_train). This involves learning the parameters (e.g., mean, standard deviation) from the training data alone.
Transform both the training data (X_train) and the testing data (X_test) using the fitted transformer.

"This ensures that the test set remains completely unseen during the parameter-learning phase of preprocessing, mirroring a scenario where you would apply transformations learned from past data to new incoming data. Scikit-learn Pipelines, which you'll encounter later in this chapter, are designed to help manage this workflow correctly and conveniently."

Exception: Time Series Data It's worth noting that for time-dependent data (time series), random shuffling and splitting are usually inappropriate. You typically want to train on older data and test on newer data to simulate forecasting future values. Specific time-series cross-validation techniques exist for these scenarios.

Splitting your data correctly is a foundational practice for evaluating machine learning models reliably. Using train_test_split with appropriate parameters like random_state and stratify provides a way to create the necessary training and testing sets for developing and assessing your models.

References

sklearn.model_selection.train_test_split, Scikit-learn developers, 2023 - Official documentation for the train_test_split function, detailing its parameters and usage.
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, Aurélien Géron, 2022 (O'Reilly Media) - A practical guide covering data splitting, overfitting, and model generalization with Python and Scikit-learn.
An Introduction to Statistical Learning: With Applications in R, Gareth James, Daniela Witten, Trevor Hastie, Rob Tibshirani, 2021 (Springer) - Explains the statistical reasoning behind data splitting for model evaluation and generalization.
Pipelining: chaining estimators, Scikit-learn developers, 2023 - Describes how Scikit-learn Pipelines prevent data leakage during preprocessing and model training.