Hypothesis Testing for Distributional Similarity

While comparing marginal distributions and basic statistics offers a first look at fidelity, it falls short of verifying if the synthetic data truly captures the relationship between variables present in the real data. We need methods that assess the similarity of the joint probability distributions, $P_{real}(X_1, ..., X_n)$ and $P_{synthetic}(X_1, ..., X_n)$. Hypothesis testing provides a formal statistical framework for this assessment.

In this context, the typical setup involves:

• Null Hypothesis ($H_0$): The real and synthetic datasets are drawn from the same underlying probability distribution. ($P_{real} = P_{synthetic}$)
• Alternative Hypothesis ($H_A$): The real and synthetic datasets are drawn from different probability distributions. ($P_{real} \neq P_{synthetic}$)

The goal of the test is to determine if there is sufficient statistical evidence to reject $H_0$ in favor of $H_A$. A small p-value (typically below a predefined significance level, e.g., 0.05) suggests that the observed differences between the datasets are unlikely to have occurred by chance if they truly came from the same distribution, leading us to reject $H_0$ and conclude the distributions are different.

Several statistical tests can be adapted or specifically designed for comparing two multivariate samples. Let's examine some prominent techniques suitable for synthetic data evaluation.

Extending Classical Tests: Challenges in High Dimensions

Classical tests like the Kolmogorov-Smirnov (KS) test and the Chi-Squared ($\chi^2$) test are well-established for univariate distribution comparison.

• Kolmogorov-Smirnov (KS) Test: The standard 1D KS test compares the cumulative distribution functions (CDFs) of two samples. Its test statistic measures the maximum absolute difference between the empirical CDFs. While powerful for univariate data, extending it directly to multiple dimensions ($>1$) is problematic. Defining a unique multivariate CDF and calculating the maximum difference becomes complex, and the tests often lose statistical power rapidly as dimensionality increases. Practical approaches sometimes involve applying 1D KS tests to projections of the data (e.g., principal components) or marginal distributions, but this doesn't fully capture the joint structure.

• Chi-Squared ($\chi^2$) Test: The Chi-Squared goodness-of-fit test or the test for homogeneity can compare distributions by partitioning the data space into bins and comparing the observed frequencies of data points falling into each bin. However, its effectiveness diminishes significantly in higher dimensions due to the "curse of dimensionality". To maintain a reasonable number of samples per bin, the number of bins required grows exponentially with the number of features, making it impractical for most multivariate datasets unless dimensionality is very low or data is inherently categorical. The choice of binning strategy also heavily influences the results.

Given these limitations, methods specifically designed for multivariate comparisons are often preferred for evaluating synthetic data fidelity.

Maximum Mean Discrepancy (MMD)

Maximum Mean Discrepancy (MMD) is a non-parametric metric that measures the distance between distributions based on the mean embeddings of samples in a Reproducing Kernel Hilbert Space (RKHS).

The intuition is to map the data points from each distribution into a potentially infinite-dimensional feature space using a kernel function $k(\cdot, \cdot)$ (e.g., Gaussian RBF kernel). MMD then calculates the squared distance between the means of the mapped samples in this RKHS.

Let $X = \{x_1, ..., x_m\}$ be samples from $P_{real}$ and $Y = \{y_1, ..., y_n\}$ be samples from $P_{synthetic}$. An empirical estimate of the squared MMD is given by:

$$MMD^2(X, Y) = \frac{1}{m^2} \sum_{i=1}^m \sum_{j=1}^m k(x_i, x_j) - \frac{2}{mn} \sum_{i=1}^m \sum_{j=1}^n k(x_i, y_j) + \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(y_i, y_j)$$

If $P_{real} = P_{synthetic}$, then $MMD(P_{real}, P_{synthetic}) = 0$. Larger values indicate greater dissimilarity.

Hypothesis Testing with MMD: A hypothesis test can be constructed using the calculated MMD value as the test statistic. The significance (p-value) is often determined using permutation testing:

1. Calculate $MMD^2(X, Y)$ for the original samples.
2. Pool the real and synthetic data.
3. Repeatedly shuffle the pooled data and split it into two new samples of sizes $m$ and $n$. Calculate $MMD^2$ for these permuted samples.
4. The p-value is the proportion of permutations where the $MMD^2$ was greater than or equal to the original $MMD^2(X, Y)$.

MMD is advantageous because it doesn't require binning, works well in high dimensions, and can capture complex differences depending on the chosen kernel. The choice of kernel and its parameters (e.g., the bandwidth $\gamma$ for an RBF kernel) is important and can affect sensitivity.

Energy Distance

Energy distance provides another powerful non-parametric framework for testing multivariate distributional equality. It's based on the statistical energy concept, related to Newton's gravitational potential energy. The E-statistic compares the distances between points from the two different samples to the distances within each sample.

Let $X = \{x_1, ..., x_m\}$ from $P_{real}$ and $Y = \{y_1, ..., y_n\}$ from $P_{synthetic}$. The E-statistic (related to squared energy distance) is calculated based on Euclidean distances $|| \cdot ||$:

$$E(X, Y) = \frac{2}{mn} \sum_{i=1}^m \sum_{j=1}^n ||x_i - y_j|| - \frac{1}{m^2} \sum_{i=1}^m \sum_{j=1}^m ||x_i - x_j|| - \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n ||y_i - y_j||$$

Similar to MMD, $E(X, Y) \ge 0$, and $E(X, Y) = 0$ if and only if $P_{real} = P_{synthetic}$ (under moment conditions). Larger values imply greater distributional differences.

Hypothesis Testing with Energy Distance: Permutation testing is also commonly used here. The E-statistic $E(X, Y)$ is calculated for the original samples, and its value is compared against the distribution of E-statistics obtained from permuted samples to compute a p-value.

Energy distance is rotation invariant and doesn't require kernel selection, making it somewhat simpler to apply than MMD in some cases, while still being effective in high dimensions.

Classifier Two-Sample Tests (C2ST)

A very intuitive approach to testing if two samples come from the same distribution is to see if a machine learning classifier can reliably distinguish between them. This is the core idea behind Classifier Two-Sample Tests (C2ST).

Methodology:

1. Labeling: Assign a label (e.g., 0) to all data points in the real dataset ($X$) and another label (e.g., 1) to all data points in the synthetic dataset ($Y$).
2. Combine & Shuffle: Combine the datasets into a single dataset with features and labels. Shuffle this combined dataset.
3. Train/Test Split: Split the combined dataset into training and testing sets.
4. Train Classifier: Train a binary classification model (e.g., Logistic Regression, SVM, Random Forest, Gradient Boosting) on the training set to predict the label (real vs. synthetic) based on the features.
5. Evaluate: Evaluate the classifier's performance on the held-out test set using a suitable metric like Accuracy or Area Under the ROC Curve (AUC).

Hypothesis Testing with C2ST: The classifier's performance on the test set serves as the test statistic.

• If $P_{real} = P_{synthetic}$, the features provide no information about the label, and the classifier should perform poorly, ideally close to random guessing (Accuracy ≈ 0.5, AUC ≈ 0.5).
• If $P_{real} \neq P_{synthetic}$, a sufficiently powerful classifier should be able to distinguish the samples, achieving performance significantly better than random chance (Accuracy > 0.5, AUC > 0.5).

The p-value can be estimated using permutation tests on the labels or by analyzing the distribution of the performance metric under the null hypothesis. A high accuracy or AUC significantly greater than 0.5 provides evidence to reject $H_0$.

Diagram illustrating the workflow of a Classifier Two-Sample Test (C2ST). Real and synthetic data are labeled, combined, split, and used to train and evaluate a classifier. High classification performance suggests the distributions are different.

C2ST is flexible as various classifiers can be used. Furthermore, if the classifier performs well, feature importance analysis can sometimes provide insights into which features or interactions differ most significantly between the real and synthetic datasets.

Practical Notes and Interpretation

When using hypothesis tests for distributional similarity:

• Test Selection: The choice depends on data characteristics (dimensionality, sample size, continuous/categorical features) and computational cost. MMD, Energy Distance, and C2ST are generally good choices for moderate-to-high dimensional continuous or mixed data.
• Significance Level ($\alpha$): Choose a significance level (e.g., $\alpha = 0.05$) beforehand. If the p-value is less than $\alpha$, you reject $H_0$.
• Interpreting p-values: Remember that a p-value is the probability of observing data at least as extreme as what was measured, assuming the null hypothesis is true.
  • A small p-value ($< \alpha$) provides evidence against $H_0$, suggesting the distributions are different.
  • A large p-value ($\ge \alpha$) means you fail to reject $H_0$. This does not prove the distributions are identical, only that the test didn't find sufficient evidence to conclude they are different at the chosen significance level with the given data. Lack of evidence isn't evidence of absence.
• Sample Size: Hypothesis tests become more sensitive with larger sample sizes. With very large datasets, even minor, practically irrelevant differences between $P_{real}$ and $P_{synthetic}$ can lead to statistically significant p-values (rejection of $H_0$). Therefore, always consider the magnitude of the difference (effect size), often represented by the test statistic itself (e.g., the MMD value, E-statistic, or classifier AUC), alongside the p-value. A statistically significant result might not be practically significant.
• Context Matters: Relate the findings back to the intended use case. Does a statistically significant difference detected by a sensitive test actually impact the downstream utility or privacy goals? A C2ST AUC of 0.6 might be statistically significant with enough data but could indicate that the datasets are still quite similar for practical purposes.

Hypothesis tests offer a rigorous way to move past simple statistics and assess the fidelity of the joint distribution. They provide quantitative evidence but should be interpreted carefully, considering sample size, effect size, and the overall evaluation goals.