Practice: Calculating Classification Metrics

Accuracy, precision, recall, and the F1-score are important metrics for evaluating classification models. Calculating these metrics is fundamental to understanding how well your model is performing. A specific example demonstrates how these values are derived from the basic counts of correct and incorrect predictions.

Scenario: Email Spam Detection

Imagine we've built a machine learning model to classify emails as either 'Spam' (the positive class) or 'Not Spam' (the negative class). We test this model on a set of 100 emails it hasn't seen before. After running the predictions and comparing them to the actual labels, we get the following results:

• True Positives (TP): 15 emails were correctly identified as Spam.
• False Positives (FP): 10 emails were incorrectly identified as Spam (they were actually 'Not Spam').
• True Negatives (TN): 70 emails were correctly identified as 'Not Spam'.
• False Negatives (FN): 5 emails were incorrectly identified as 'Not Spam' (they were actually 'Spam').

Let's verify the total number of emails: $15 (TP) + 10 (FP) + 70 (TN) + 5 (FN) = 100$ emails. This matches our test set size.

Step 1: Construct the Confusion Matrix

First, let's organize these results into the confusion matrix format we learned about earlier. Remember, rows typically represent the actual class, and columns represent the predicted class.

	Predicted: Spam	Predicted: Not Spam	Total Actual
Actual: Spam	TP = 15	FN = 5	20
Actual: Not Spam	FP = 10	TN = 70	80
Total Predicted	25	75	100

This matrix gives us a clear visual summary of the model's performance. We can see the number of correct predictions along the diagonal (TP and TN) and the errors off the diagonal (FP and FN). We also see the total actual spam (20) and not spam (80), as well as how many the model predicted as spam (25) and not spam (75).

Step 2: Calculate Accuracy

Accuracy tells us the overall proportion of correct predictions.

The formula is:

$$Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$$

Plugging in our values:

$$Accuracy = \frac{15 + 70}{15 + 10 + 70 + 5} = \frac{85}{100} = 0.85$$

So, the model's accuracy is 85%. This means it correctly classified 85 out of the 100 emails. While 85% sounds good, we know accuracy can sometimes be misleading, especially if the classes are imbalanced (here, we have 80 'Not Spam' vs. 20 'Spam'). Let's calculate other metrics for a more complete picture.

Step 3: Calculate Precision

Precision measures the accuracy of the positive predictions. Out of all emails the model predicted as Spam, how many actually were Spam?

The formula is:

$$Precision = \frac{TP}{TP + FP}$$

Using our values from the confusion matrix (look at the 'Predicted: Spam' column):

$$Precision = \frac{15}{15 + 10} = \frac{15}{25} = 0.60$$

The precision is 60%. This tells us that when our model flags an email as Spam, it is correct 60% of the time. The remaining 40% are False Positives (legitimate emails incorrectly marked as spam).

Step 4: Calculate Recall (Sensitivity)

Recall (also called Sensitivity or True Positive Rate) measures how many of the actual positive cases the model correctly identified. Out of all the emails that actually were Spam, how many did the model find?

The formula is:

$$Recall = \frac{TP}{TP + FN}$$

Using our values from the confusion matrix (look at the 'Actual: Spam' row):

$$Recall = \frac{15}{15 + 5} = \frac{15}{20} = 0.75$$

The recall is 75%. This means our model successfully identified 75% of all the actual Spam emails in the test set. The remaining 25% were False Negatives (spam emails that slipped through the filter).

Step 5: Calculate F1-Score

The F1-score provides a single metric that balances Precision and Recall, using their harmonic mean. This is useful when we want a measure that considers both types of errors (FP and FN).

The formula is:

$$F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}$$

Using the Precision (0.60) and Recall (0.75) we just calculated:

$$F1 = 2 \times \frac{0.60 \times 0.75}{0.60 + 0.75} = 2 \times \frac{0.45}{1.35} = \frac{0.90}{1.35} \approx 0.667$$

The F1-score is approximately 66.7%. This single number gives us a combined sense of the model's performance regarding precision and recall.

Summary of Results

Let's visualize these important metrics:

Calculated performance metrics for the spam detection example.

Interpretation

By calculating these metrics, we gain much more insight than just looking at the 85% accuracy:

• Accuracy (85%): Overall correctness is high.
• Precision (60%): When the model says "Spam", it's right 6 out of 10 times. This means 4 out of 10 times, it flags a legitimate email as spam (False Positive). This might annoy users.
• Recall (75%): The model catches 3 out of 4 actual spam emails. This means 1 out of 4 spam emails gets through to the inbox (False Negative). This might expose users to unwanted spam.
• F1-Score (66.7%): Provides a balanced measure, useful for comparing models, especially when precision and recall have different values.

Notice the trade-off. If we adjusted the model to be more aggressive in flagging spam (potentially increasing Recall), we might also increase False Positives, thus lowering Precision. Conversely, making the model more conservative to avoid flagging legitimate emails (increasing Precision) might let more actual spam through (lowering Recall). The relative importance of Precision versus Recall often depends on the specific application. For spam detection, users might tolerate some spam getting through (lower Recall) more than having important emails flagged as spam (requiring higher Precision).

This practice exercise demonstrates how calculating these standard metrics from the basic TP, FP, TN, and FN counts allows for a much richer understanding of a classification model's behavior and its suitability for a given task.