Basic Statistical Comparisons

For an objective comparison of data, quantitative measures are often necessary. Basic statistical comparisons provide hard numbers for this purpose. They offer a quantitative way to check if the fundamental properties of individual features (like columns in a table) in your synthetic data match those in the real data.

Think of these statistics as simple summaries. Just like you might summarize a long book with a few sentences, statistics like the mean, median, or standard deviation summarize important aspects of your data columns. By comparing these summaries between your real and synthetic datasets, you get a quick numerical check on how similar they are at a basic level.

Comparing Core Tendencies and Spread

The most common statistics to compare are measures of central tendency and dispersion.

• Mean ($\mu$): This is the average value of a feature. Comparing the mean tells you if, on average, the values in your synthetic feature are centered around the same point as the real feature. For example, if the average age in your real customer data is 45.2, you'd hope the average age in your synthetic data is close to that.
• Median: This is the middle value when the data is sorted. It's less sensitive to extreme outliers than the mean. Comparing medians helps check if the central point is similar, especially if your data has some very large or very small values.
• Standard Deviation ($\sigma$): This measures how spread out the data is around the mean. A small standard deviation means data points are clustered closely around the average, while a large one indicates they are more spread out. Comparing standard deviations tells you if your synthetic data has a similar level of variability as the real data.
• Variance ($\sigma^2$): This is simply the square of the standard deviation. It also measures spread but in squared units, which is sometimes useful mathematically. Comparing variance achieves a similar goal to comparing standard deviation.
• Minimum and Maximum: Comparing the smallest and largest values for a feature helps ensure the synthetic data stays within realistic bounds observed in the original data.

How to Perform the Comparison

The process is straightforward:

1. Choose a Feature: Select a column (feature) you want to evaluate, usually a numerical one for these statistics (e.g., 'Age', 'Income', 'Temperature').
2. Calculate Statistics: Compute the desired statistic (e.g., mean, standard deviation) for this feature using the real dataset.
3. Calculate Statistics Again: Compute the same statistic for the corresponding feature in the synthetic dataset.
4. Compare: Look at the two numbers. How close are they?

For instance, imagine you have real customer data and generated synthetic customer data. Let's look at the 'Age' column:

• Real Data: Mean Age = 42.3 years, Std Dev Age = 10.5 years
• Synthetic Data: Mean Age = 41.9 years, Std Dev Age = 10.8 years

In this case, the means are very close (42.3 vs 41.9), and the standard deviations are also quite similar (10.5 vs 10.8). This suggests that, for the 'Age' feature, the synthetic data generation process did a reasonable job of capturing the average value and the typical spread around that average.

What if the synthetic data showed a Mean Age of 35.1 years? That significant difference would immediately signal a problem. The synthetic data isn't centered correctly for age compared to the real data. Similarly, if the synthetic Standard Deviation was 2.5 years, it would indicate the synthetic ages are unrealistically clustered near the mean, lacking the diversity of the real data.

Visualizing Statistical Differences

Comparing numbers pair by pair works, but it can be tedious if you have many features. A common approach is to calculate these basic statistics for all relevant features in both datasets and then plot them side-by-side for easier comparison. Bar charts are often used for this purpose.

Let's say we compared the mean for three features: 'Age', 'Income' (in $1000s), and 'Years_Customer'.

Comparison of mean values for selected features between the real and synthetic datasets.

We can do the same for standard deviation:

Comparison of standard deviation for selected features between the real and synthetic datasets.

These charts allow you to quickly spot features where the basic statistics differ substantially between the real and synthetic data. In the examples above, 'Income' shows a larger relative difference in both mean and standard deviation compared to 'Age' or 'Years_Customer', suggesting the synthesis might be less accurate for that particular feature.

Important Caveats

Matching basic statistics like mean and standard deviation is a good first check, but it's not a guarantee that the synthetic data is a perfect replica. It's possible for two datasets to have identical means and standard deviations but possess very different shapes or distributions. Think about it: a dataset with values clustered at both ends could have the same average as one where values are all clustered in the middle.

Therefore, these basic statistical comparisons are a necessary, but not sufficient, step in evaluation. They tell you if the synthetic data is centered correctly and has a similar overall spread for individual features, but they don't capture the full picture of the data's structure or the relationships between features. We need to combine this analysis with visual inspection and methods that compare the entire data distribution, which we'll discuss next.