Chapter 3: Evaluating Machine Learning Utility

After assessing the statistical similarity between real and synthetic datasets, we turn to a critical practical question: How useful is the synthetic data for training actual machine learning models? This chapter focuses on quantifying this machine learning utility.

You will learn standard evaluation approaches, primarily the Train-Synthetic-Test-Real (TSTR) methodology. In TSTR, a model is trained using only synthetic data and then evaluated on a held-out set of real data. We will also examine the complementary Train-Real-Test-Synthetic (TRTS) approach.

Key techniques covered include:

• Comparing standard performance metrics (like accuracy, F1-score, or AUC) between models trained on real versus synthetic data.
• Analyzing whether models trained on synthetic data learn similar feature importance patterns as those trained on real data.
• Understanding how using synthetic data might influence the outcome of hyperparameter tuning.

The goal is to provide objective measures to determine if the synthetic data can effectively stand in for real data in your downstream machine learning applications. This involves hands-on practice implementing these evaluation workflows.