Information-Theoretic Measures

While comparing marginal distributions and basic statistics offers a first glance at fidelity, as discussed earlier, these methods often miss the intricate web of relationships between variables within a dataset. Information theory provides a powerful mathematical framework to quantify uncertainty and dependence, offering more sophisticated tools to compare the structural properties of real ($D_{real}$) and synthetic ($D_{synth}$) datasets.

Instead of just asking "Are the means similar?", information-theoretic measures help us ask "Does the synthetic data contain the same amount of information?" or "Are the relationships between variables preserved in the synthetic data?".

Entropy: Quantifying Uncertainty

Entropy is a fundamental measure of the uncertainty or randomness associated with a random variable. For a discrete random variable $X$ with possible values $x_1, ..., x_k$ and probability mass function $P(X)$, the Shannon entropy $H(X)$ is defined as:

$$H(X) = - \sum_{i=1}^{k} P(x_i) \log_2 P(x_i)$$

The logarithm base is typically 2, measuring entropy in bits. Higher entropy implies more uncertainty or less predictability.

In synthetic data evaluation:

1. Marginal Entropy Comparison: You can compute the entropy for each feature $X_i$ in both $D_{real}$ and $D_{synth}$. Significant differences in $H(X_{i, real})$ versus $H(X_{i, synth})$ indicate that the synthetic data generation process has altered the inherent variability or predictability of that feature. For continuous variables, differential entropy is used, often requiring techniques like discretization (binning) or kernel density estimation (KDE) followed by numerical integration, which introduces approximation challenges.
2. Joint Entropy Comparison: More powerfully, you can estimate the joint entropy $H(X_1, ..., X_n)$ for subsets of features (or ideally, all features) in both datasets. This measures the total uncertainty associated with the group of variables considered together. Differences in joint entropy suggest that the overall complexity or randomness of the multivariate system is not well captured. However, accurately estimating high-dimensional joint entropy is computationally demanding and suffers from the curse of dimensionality.

Mutual Information: Quantifying Shared Information

Mutual Information (MI) measures the amount of information obtained about one random variable by observing another. It quantifies the dependency between two variables, capturing non-linear relationships beyond simple correlation. For two discrete variables $X$ and $Y$, the mutual information $I(X; Y)$ is:

$$I(X; Y) = \sum_{x \in X} \sum_{y \in Y} P(x, y) \log_2 \left( \frac{P(x, y)}{P(x)P(y)} \right)$$

It can also be expressed using entropy:

$$I(X; Y) = H(X) + H(Y) - H(X, Y)$$

$I(X; Y)$ is always non-negative and is zero if and only if $X$ and $Y$ are independent. A higher MI indicates stronger dependence.

Application in Evaluation: A key aspect of statistical fidelity is preserving the relationships between variables. We can compute the MI for pairs of features $(X_i, X_j)$ in both the real and synthetic datasets: $I(X_{i, real}; X_{j, real})$ vs $I(X_{i, synth}; X_{j, synth})$.

• If the MI values are consistently similar across many pairs of features, it suggests the synthetic data generation process successfully learned and replicated the pairwise dependencies present in the real data.
• Large discrepancies, especially for pairs known to be strongly related in the real data, signal a failure to capture important structural information.

Calculating MI typically requires estimating the joint and marginal probability distributions. For continuous data, this often involves discretization or using specialized estimators like k-nearest neighbors (KNN) based methods available in libraries such as scikit-learn (sklearn.metrics.mutual_info_score for discrete, sklearn.feature_selection.mutual_info_regression/mutual_info_classification which use KNN estimation for continuous/mixed types).

import numpy as np
from sklearn.metrics import mutual_info_score
from sklearn.preprocessing import KBinsDiscretizer

# Assume real_data and synth_data are pandas DataFrames
# Example: Compare MI between 'Age' and 'Income' (assuming they are continuous)

# Discretize continuous features for MI calculation using mutual_info_score
n_bins = 10 # Choose an appropriate number of bins
kbd = KBinsDiscretizer(n_bins=n_bins, encode='ordinal', strategy='uniform', subsample=None)

# Fit on real data and transform both real and synthetic
# Important: Fit the discretizer ONLY on the real data
real_data_discrete = kbd.fit_transform(real_data[['Age', 'Income']])
synth_data_discrete = kbd.transform(synth_data[['Age', 'Income']])

# Calculate contingency matrix (necessary for mutual_info_score)
# Note: mutual_info_score expects labels, so we use the discretized values
# We need the joint distribution P(x,y), often estimated via histogramming / contingency table

def calculate_mi(data_col1, data_col2):
    """ Calculates MI between two discrete columns """
    contingency_matrix = np.histogram2d(data_col1, data_col2, bins=n_bins)[0]
    # Add a small epsilon to avoid log(0) if necessary, or handle appropriately
    # mutual_info_score handles this internally
    return mutual_info_score(None, None, contingency_matrix=contingency_matrix)

mi_real = calculate_mi(real_data_discrete[:, 0], real_data_discrete[:, 1])
mi_synth = calculate_mi(synth_data_discrete[:, 0], synth_data_discrete[:, 1])

print(f"Mutual Information (Real Data, Age vs Income): {mi_real:.4f}")
print(f"Mutual Information (Synth Data, Age vs Income): {mi_synth:.4f}")
print(f"Difference in MI: {abs(mi_real - mi_synth):.4f}")

# Consider doing this for multiple feature pairs

Code snippet demonstrating the comparison of Mutual Information between two features ('Age', 'Income') after discretization in real and synthetic datasets using scikit-learn.

Divergence Measures: Quantifying Distributional Difference

Divergence measures provide a way to quantify the difference between two probability distributions, $P$ (representing the real data distribution) and $Q$ (representing the synthetic data distribution).

1. Kullback-Leibler (KL) Divergence: Measures how much information is lost when approximating $P$ with $Q$. For discrete distributions:

   $$D_{KL}(P||Q) = \sum_{x} P(x) \log_2 \left( \frac{P(x)}{Q(x)} \right)$$

   $D_{KL}(P||Q) \ge 0$, with $D_{KL}(P||Q) = 0$ if and only if $P=Q$.

   • Properties: Asymmetric ($D_{KL}(P||Q) \neq D_{KL}(Q||P)$). Undefined if $Q(x)=0$ when $P(x) > 0$.
   • Use: Can compare marginal distributions ($P(X_i)$ vs $Q(X_i)$) or, more importantly, joint distributions ($P(X_1,...,X_n)$ vs $Q(X_1,...,X_n)$). Lower KL divergence indicates better fidelity.

2. Jensen-Shannon (JS) Divergence: A symmetric and bounded measure derived from KL divergence.

   $$JSD(P||Q) = \frac{1}{2} D_{KL}(P||M) + \frac{1}{2} D_{KL}(Q||M)$$

   where $M = \frac{1}{2}(P+Q)$ is the mixture distribution.

   • Properties: Symmetric ($JSD(P||Q) = JSD(Q||P)$). Bounded between 0 and 1 (when using log base 2). Always defined if $P$ and $Q$ are defined over the same support.
   • Use: Often preferred over KL divergence for comparing distributions due to its symmetry and finite range. A JS divergence close to 0 indicates high similarity between the real and synthetic distributions, while a value closer to 1 indicates high divergence.

Practical Calculation: Calculating divergences for high-dimensional joint distributions is challenging. Common approaches involve:

• Discretization: Bin the continuous feature space and estimate $P$ and $Q$ as histograms. The choice of binning strategy significantly impacts the result.
• Kernel Density Estimation (KDE): Estimate smooth probability density functions $p(x)$ and $q(x)$ and compute the divergence via numerical integration. Requires careful bandwidth selection.
• Subspace Evaluation: Calculate average divergences over randomly selected low-dimensional marginals (e.g., pairs or triplets of features) as a proxy for the full joint distribution difference.

Libraries like scipy.stats.entropy can compute KL divergence between discrete distributions (represented as probability vectors). JS divergence can be implemented using the KL divergence function.

from scipy.stats import entropy
import numpy as np

# Assume p_real and q_synth are 1D probability distributions (histograms)
# e.g., from np.histogram(data_feature, bins=..., density=True)[0]
# Ensure they sum to 1 and have the same bins/support. Add small epsilon to avoid zeros.

epsilon = 1e-10
p_real = p_real + epsilon
p_real /= np.sum(p_real)
q_synth = q_synth + epsilon
q_synth /= np.sum(q_synth)

# KL Divergence (Real || Synth)
kl_div_pq = entropy(p_real, q_synth, base=2)

# KL Divergence (Synth || Real)
kl_div_qp = entropy(q_synth, p_real, base=2)

# JS Divergence
m = 0.5 * (p_real + q_synth)
js_div = 0.5 * (entropy(p_real, m, base=2) + entropy(q_synth, m, base=2))

print(f"KL Divergence (P||Q): {kl_div_pq:.4f}")
print(f"KL Divergence (Q||P): {kl_div_qp:.4f}")
print(f"JS Divergence (P||Q): {js_div:.4f}") # Should be between 0 and 1

Basic calculation of KL and JS divergence between two 1D probability distributions using SciPy. Requires careful preprocessing (normalization, handling zeros).

Summary and Considerations

Information-theoretic measures offer a theoretically grounded way to assess statistical fidelity beyond simple statistics:

• Entropy quantifies randomness and complexity.
• Mutual Information specifically targets the preservation of inter-variable dependencies, including non-linear ones.
• KL and JS Divergences provide overall measures of distributional similarity.

Advantages:

• Capture complex dependencies and distributional shapes.
• Provide principled, quantitative scores.
• JS divergence is symmetric and bounded, aiding interpretation.

Challenges:

• Computationally intensive, especially for high dimensions and large datasets.
• Sensitive to estimation methods (discretization parameters, KDE bandwidth).
• Estimating high-dimensional joint distributions reliably is difficult (curse of dimensionality).

Despite the challenges, comparing mutual information between feature pairs and using JS divergence on lower-dimensional marginals (or carefully estimated joint distributions) provide valuable insights into how well the synthetic data replicates the statistical structure of the real data, significantly complementing other fidelity assessment techniques.