Common Mistakes in Basic Evaluation

Executing the evaluation workflow seems straightforward enough: split the data, train the model, make predictions, and calculate metrics. However, several common errors can undermine your results, leading you to trust models that won't perform well in the real world or discard models that are actually quite useful. Let's look at some frequent missteps to avoid.

Evaluating on Training Data

This is perhaps the most fundamental error. After training a model on a dataset, it becomes very good at predicting the outcomes for that specific data. If you then evaluate the model using the same data it learned from, you'll likely get highly optimistic performance scores.

Think of it like giving students the exact same questions for their final exam that they used to study. They might score 100%, but does that mean they truly understand the subject or just memorized the answers? No. The model essentially "memorizes" the training data, including its noise and quirks. This phenomenon is called overfitting.

Consequence: You get a misleadingly high performance score, making you think the model is excellent, when it might fail significantly on new, unseen data.

Solution: Always evaluate your model on a separate test set that was not used during training. This provides a much more realistic estimate of how the model will generalize to new situations.

Leaking Data During Preprocessing

Data leakage occurs when information from outside the training dataset is inadvertently used to create the model. A common way this happens is during data preprocessing steps like scaling (e.g., standardization or normalization) or imputation (filling missing values).

Imagine you have a dataset with missing values. If you calculate the mean of a feature using the entire dataset (including the test set) and then use that mean to fill missing values in both the training and test sets before splitting, you've leaked information. The training process has implicitly learned something about the test set's distribution via that calculated mean. Similarly, if you scale features based on the minimum and maximum values found in the entire dataset before splitting, the training data scaling is influenced by the test data.

Consequence: Like evaluating on the training set, data leakage leads to overly optimistic performance estimates because the model implicitly "knows" something about the test data it shouldn't.

Solution: Perform the train-test split first. Then, fit your preprocessors (like scalers or imputers) only on the training data. Use the fitted preprocessor to transform both the training data and the test data. This ensures that the test set remains completely unseen during the entire model building process, including preprocessing.

# Incorrect approach (potential leakage)
1. Load full dataset
2. Scale features using min/max of full dataset
3. Split into train/test
4. Train model on scaled train data
5. Evaluate on scaled test data

# Correct approach (no leakage)
1. Load full dataset
2. Split into train/test
3. Fit scaler on train data only (find min/max from train)
4. Scale train data using the fitted scaler
5. Scale test data using the *same* fitted scaler
6. Train model on scaled train data
7. Evaluate on scaled test data

Choosing the Wrong Metric

Not all metrics are created equal, and the best metric depends heavily on the problem you're trying to solve and the characteristics of your data.

• Classification: Using only accuracy on an imbalanced dataset is a classic example. If 99% of your data belongs to Class A and only 1% to Class B, a model that always predicts Class A will achieve 99% accuracy! This sounds great, but the model is useless for identifying Class B. In such cases, precision, recall, F1-score, or metrics like the AUC (Area Under the Curve, covered in more advanced topics) are much more informative.
• Regression: While R-squared tells you the proportion of variance explained, it doesn't tell you the magnitude of the errors. A model predicting house prices could have a high R-squared but still be off by an average of $50,000 (MAE), which might be unacceptable. Conversely, MAE or RMSE tell you the average error magnitude but not necessarily how much better the model is than simply predicting the average price.

Consider a medical diagnosis task where identifying a disease (positive class) is critical.

• High Recall: You want to find as many actual patients as possible, even if you misclassify some healthy people (minimize False Negatives).
• High Precision: You want to be very sure that when the model predicts someone has the disease, they actually do (minimize False Positives).

Consequence: You might optimize for a metric that doesn't reflect the actual goal, leading to a model that performs poorly in practice according to what truly matters.

Solution: Understand the implications of different types of errors for your specific problem. Choose metrics that align with your objectives. For classification, always examine the confusion matrix, especially with imbalanced data. For regression, consider both error magnitude (MAE, RMSE) and relative performance (R-squared).

Forgetting the Baseline

Before celebrating your model's performance, always compare it to a simple baseline. A baseline is a very basic model or rule that provides a point of reference.

• Classification Baseline: Predict the most frequent class for all instances.
• Regression Baseline: Predict the mean (or median) of the target variable for all instances.

If your sophisticated machine learning model doesn't perform significantly better than this simple baseline, it might be overly complex, poorly tuned, or the features might not contain enough predictive information.

Consequence: You might invest time and resources developing a complex model that offers little or no practical advantage over a trivial approach.

Solution: Always calculate the performance of a simple baseline model using the same metrics you use for your actual model. Use this as a benchmark to judge whether your model provides meaningful improvement.

Misinterpreting Metric Values

Getting a number from a metric is easy; understanding what it means requires context.

• Accuracy: As mentioned, 99% accuracy can be terrible if the data is imbalanced.
• Precision/Recall: A model might have high precision but low recall, or vice-versa. Understanding the trade-off (discussed in Chapter 2) is essential. Simply looking at one without the other can be misleading.
• RMSE: An RMSE of 10 might be excellent if you're predicting values ranging from 0 to 1000, but terrible if the range is 0 to 20. The units and scale matter.
• R-squared: A high R-squared doesn't guarantee the model is good or that its predictions are accurate in an absolute sense. It measures the proportion of variance explained relative to the mean. It can be artificially inflated by adding more features, even useless ones (adjusted R-squared, a more advanced topic, tries to correct for this).

Consequence: Making poor decisions based on a superficial understanding of the performance scores. You might deploy a model believing it's good when its practical performance is inadequate.

Solution: Don't just report the numbers. Interpret them in the context of your specific problem, the data's scale and distribution, and the baseline performance. Understand the definition and limitations of each metric you use.

Relying on a Single Data Split

As briefly touched upon in the previous chapter, splitting your data once into a training set and a test set gives you one estimate of performance. However, this estimate can be sensitive to exactly how the split was made. By chance, you might get an "easy" test set where your model performs unusually well, or an unusually "hard" one where it performs poorly.

Consequence: Your single evaluation might not be representative of how the model would perform on average across different subsets of unseen data. The results might be too optimistic or too pessimistic purely due to luck of the draw in the split.

Solution: While a single train-test split is fundamental for basic evaluation, be aware of this limitation. More robust evaluation techniques like cross-validation (introduced conceptually in Chapter 4 and covered in detail in more advanced courses) involve multiple splits and averaging the results to get a more stable performance estimate. For now, recognize that a single split provides an estimate, not an exact measure of future performance.

Avoiding these common mistakes is fundamental to reliable model evaluation. Careful attention to the evaluation process ensures that the performance metrics you calculate genuinely reflect your model's ability to generalize and provide value on new, unseen data.