Taxonomy of Evaluation Metrics

Evaluating synthetic data involves a multi-faceted process that considers statistical similarity, practical usefulness, and privacy assurance. The variety of potential checks, ranging from simple statistical comparisons to complex attack simulations, necessitates organization for a systematic evaluation. Metrics can be structured by the type of assessment they perform, which generally aligns with the core quality dimensions: fidelity, utility, and privacy.

We can broadly group evaluation metrics into the following categories:

Statistical Fidelity Metrics

These metrics quantify how closely the statistical properties of the synthetic dataset ( $D_{syn}$ ) mirror those of the real dataset ( $D_{real}$ ). They directly assess the "likeness" of the generated data distribution to the original one.

Marginal Distribution Comparisons: These are often the first step, comparing the distribution of individual columns (features) between $D_{real}$ $D_{re a l}$ and $D_{syn}$ $D_{sy n}$ . Common techniques include:
- Visualizations: Overlaying histograms or density plots for each feature.
- Summary Statistics: Comparing mean, median, variance, quantiles, etc.
- Statistical Tests: Applying tests like the Kolmogorov-Smirnov (KS) test or Shapiro-Wilk test (for normality checks) to individual features. While simple, these univariate checks don't capture relationships between features.
Multivariate Distribution Comparisons: Assessing the joint distribution and inter-feature dependencies is more informative but also more complex. Methods include:
- Correlation Analysis: Comparing correlation matrices (e.g., using Frobenius norm difference) between $D_{real}$ and $D_{syn}$ .
- Dimensionality Reduction: Applying techniques like Principal Component Analysis (PCA) to both datasets and comparing the resulting principal components or the distribution of data in the reduced space.
- Advanced Statistical Tests: Utilizing multivariate hypothesis tests (e.g., Hotelling's T-squared test for means, tests based on maximum mean discrepancy or energy statistics) to compare the overall distributions. We will examine these in detail in Chapter 2.
- Propensity Scores: Training a classifier to distinguish between real and synthetic samples. A classifier performing near random chance (e.g., AUC close to 0.5) indicates high similarity.

Machine Learning Utility Metrics

These metrics evaluate the practical value of the synthetic data for training downstream machine learning models. The central question is: Can a model trained on $D_{syn}$ perform comparably to a model trained on $D_{real}$ when evaluated on unseen real data?

Train-Synthetic-Test-Real (TSTR): This is a standard protocol. Train a specific ML model (e.g., logistic regression, random forest, neural network) on $D_{syn}$ and evaluate its performance (accuracy, F1-score, AUC, etc.) on a held-out portion of $D_{real}$ . Compare this performance to a baseline model trained and tested on different portions of $D_{real}$ .
Train-Real-Test-Synthetic (TRTS): A complementary approach where a model is trained on $D_{real}$ and tested on $D_{syn}$ . This can help identify if the synthetic data contains unrealistic patterns or fails to represent certain aspects of the real data distribution that the model learned.
Downstream Model Analysis: Rather than just performance scores, utility can be assessed by comparing:
- Feature importance rankings derived from models trained on $D_{real}$ vs. $D_{syn}$ .
- Model coefficients or decision boundaries.
- Behavior during hyperparameter optimization. Chapter 3 focuses on implementing and interpreting these utility assessments.

Privacy Metrics

These metrics aim to quantify the risk of sensitive information about individuals in $D_{real}$ being leaked through $D_{syn}$ . This is distinct from fidelity and utility; highly accurate synthetic data might inadvertently pose a high privacy risk.

Membership Inference Attacks (MIAs): Attempting to determine if a specific record from $D_{real}$ was used to train the generative model by analyzing $D_{syn}$ . Success rates above random guessing indicate potential privacy leakage.
Attribute Inference Attacks: Trying to predict sensitive attributes of a real individual given some of their other attributes (present in $D_{syn}$ ) and access to the synthetic dataset.
Distance-Based Metrics: Calculating distances between synthetic records and their nearest neighbors in the real dataset. Very small distances (especially for identical matches) can signal privacy issues like data copying.
Differential Privacy (DP) Guarantees: If the generative model incorporated DP mechanisms, the privacy loss parameters (like $\epsilon$ and $\delta$ ) provide a formal, quantifiable measure of privacy protection. We will explore methods for implementing and interpreting these privacy assessments in Chapter 4.

Domain-Specific Metrics

The metrics above are largely applicable to tabular data. However, different data modalities often require specialized evaluation techniques.

Image Data: Metrics like Fréchet Inception Distance (FID), Inception Score (IS), and Precision/Recall assess the quality and diversity of generated images based on features extracted from pre-trained deep learning models.
Text Data: Evaluating generated text involves metrics like Perplexity (measuring model uncertainty), BLEU scores (comparing generated text to reference texts), and semantic similarity scores.
Time-Series Data: Assessment focuses on capturing temporal dependencies, using metrics like autocorrelation function (ACF) similarity, spectral density comparisons, and performance on downstream forecasting tasks. Chapter 5 covers these specialized metrics in greater detail.

A categorization of synthetic data evaluation metrics based on the assessment goal.

Understanding this taxonomy provides a roadmap for comprehensive evaluation. Rarely is a single metric sufficient. Instead, a suite of metrics across these categories is typically needed to gain a holistic view of the synthetic data's quality, reflecting the inherent trade-offs between fidelity, utility, and privacy that we introduced earlier. The specific choice of metrics within this framework will depend heavily on the data type, the generative model used, and the intended application of the synthetic data. In the following chapters, we will implement and interpret metrics from each of these categories.

Was this section helpful?

References

Synthetic Data - What, Why and How?, Cohen, Adrian Weller, 2022 Foundations and Trends® in Machine Learning, Vol. 16 (Now Publishers) DOI: 10.48550/ARXIV.2205.03257 - This survey provides a comprehensive overview of synthetic data generation and discusses its evaluation, covering fidelity, utility, and privacy aspects.
Membership Inference Attacks Against Machine Learning Models, Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov, 2017 2017 IEEE Symposium on Security and Privacy (SP) (IEEE) DOI: 10.1109/SP.2017.52 - Introduced membership inference attacks, a key privacy metric for assessing data leakage risk in synthetic datasets.
The Algorithmic Foundations of Differential Privacy, Cynthia Dwork and Aaron Roth, 2014 Foundations and Trends® in Theoretical Computer Science, Vol. 9 (Now Publishers) DOI: 10.1561/0400000042 - This book presents the theoretical foundations and formal guarantees of differential privacy, essential for understanding privacy metrics.