Train-Real-Test-Synthetic (TRTS) Methodology

The Train-Synthetic-Test-Real (TSTR) approach directly measures if synthetic data can replace real data for training. In contrast, the Train-Real-Test-Synthetic (TRTS) methodology offers a different perspective on synthetic data quality. TRTS evaluates how well synthetic data resembles real data from the viewpoint of a model trained on the original dataset, rather than assessing the synthetic data's training utility.

The TRTS workflow is essentially the inverse of TSTR:

1. Train on Real Data: Train a chosen machine learning model (let's call it $Model_R$) using the original, real training dataset ($Real_{train}$).
2. Test on Synthetic Data: Evaluate the performance of $Model_R$ using the synthetically generated dataset ($Synthetic_{gen}$) as the test set.
3. (Optional) Test on Real Hold-out Data: For comparison, evaluate $Model_R$ on a held-out portion of the real data ($Real_{test}$).

Here's a diagram illustrating the flow:

The TRTS evaluation process: A model is trained exclusively on real data and subsequently tested on the synthetic dataset. Performance on a real test set often serves as a baseline.

Interpreting TRTS Results

The performance metrics obtained from testing $Model_R$ on $Synthetic_{gen}$ (e.g., accuracy, AUC, F1-score) provide insights into the synthetic data's characteristics:

• High Performance on Synthetic Data: If $Model_R$ performs well on $Synthetic_{gen}$, it suggests that the patterns, relationships, and decision boundaries learned from $Real_{train}$ are also present and recognizable in the synthetic data. The synthetic data effectively mimics the distribution that $Model_R$ learned.
• Low Performance on Synthetic Data: Conversely, poor performance indicates that $Model_R$, despite being proficient on the real data it was trained on, struggles to generalize to the synthetic data. This implies a distributional mismatch; the synthetic data lacks or misrepresents the patterns $Model_R$ identified as important in the real data.

Comparing TRTS with TSTR and Baselines

Comparing the TRTS score (performance of $Model_R$ on $Synthetic_{gen}$) with the performance of the same model on the real test set ($Real_{test}$) is informative:

• TRTS Score ≈ Real Test Score: This is often a desirable outcome. It suggests the synthetic data mirrors the real data's characteristics well enough that a model trained on real data performs similarly on both. The synthetic data appears representative.
• TRTS Score > Real Test Score: This scenario might seem positive initially, but it requires careful scrutiny. It could mean the synthetic data is too similar to the specific $Real_{train}$ dataset used to train $Model_R$. This might happen if the generative model overfits or memorizes aspects of its training data. While fidelity to the training set is high, the synthetic data might lack the diversity or generalizability present in unseen real data ($Real_{test}$).
• TRTS Score < Real Test Score: This is a common result, indicating that the synthetic data doesn't capture all the nuances or the exact distribution of the real data, causing the real-data-trained model to perform worse on it.

TRTS complements TSTR by answering a different question.

• TSTR asks: "Can I train a useful model using only synthetic data?" (Focus: Replaceability)
• TRTS asks: "Does the synthetic data look statistically similar to the real data, from the perspective of a model trained on real data?" (Focus: Representativeness)

A high TSTR score indicates practical utility for model training. A high TRTS score suggests the generator successfully learned patterns from the real training set, but it doesn't guarantee TSTR utility, especially if the high score is due to overfitting during generation. Ideally, you seek synthetic data that performs well in both TSTR and TRTS evaluations relative to real-data baselines, indicating a good balance of utility and fidelity without simple memorization.

Like TSTR, the choice of the downstream model ($Model_R$) and the evaluation metrics can influence TRTS results. It's often beneficial to run TRTS using the same model architecture planned for the actual downstream task.