Evaluating Synthetic Images: FID, IS, Precision, Recall

While general-purpose statistical metrics provide a baseline for evaluation, assessing the quality of synthetic images demands specialized techniques. Human visual perception is highly attuned to subtle details, textures, and structural coherence, aspects that global statistical measures might miss. Furthermore, generative models for images, like Generative Adversarial Networks (GANs) or Diffusion Models, often have unique failure modes, such as mode collapse (lack of diversity) or unrealistic artifacts. Metrics specifically developed to evaluate the perceptual quality, fidelity, and diversity of generated images are examined.

Fréchet Inception Distance (FID)

The Fréchet Inception Distance (FID) is one of the most widely adopted metrics for evaluating the quality of synthetic images, particularly those generated by GANs. It aims to measure the similarity between the distribution of synthetic images and the distribution of real images in a feature space derived from a pre-trained image classification network.

Intuition: The core idea is that if synthetic images are realistic and diverse, their high-level features (as perceived by a powerful image classifier like Inception v3) should closely match the distribution of features extracted from real images. FID quantifies the distance between these two feature distributions.

How it Works:

1. Feature Extraction: A pre-trained Inception v3 network (typically trained on ImageNet) is used. Both a set of real images ($X$) and a set of synthetic images ($G$) are passed through the network, and activations from a specific layer (usually the final average pooling layer before classification) are extracted. These activations serve as feature vectors for each image.
2. Distribution Modeling: The collection of feature vectors for the real images and the synthetic images are each modeled as a multivariate Gaussian distribution. The mean ($\mu_x, \mu_g$) and covariance matrix ($\Sigma_x, \Sigma_g$) are calculated for each set of features.
3. Fréchet Distance Calculation: The FID score is the Fréchet distance (also known as the Wasserstein-2 distance) between these two Gaussian distributions ($N(\mu_x, \Sigma_x)$ and $N(\mu_g, \Sigma_g)$).

The formula for the Fréchet distance between two multivariate Gaussians is:

$$FID(x, g) = ||\mu_x - \mu_g||^2_2 + \text{Tr}(\Sigma_x + \Sigma_g - 2(\Sigma_x \Sigma_g)^{1/2})$$

Where:

• $||\mu_x - \mu_g||^2_2$ is the squared Euclidean distance between the mean vectors.
• $\text{Tr}$ denotes the trace of a matrix (the sum of diagonal elements).
• $(\Sigma_x \Sigma_g)^{1/2}$ is the matrix square root of the product of the covariance matrices.

Interpretation:

• Lower FID scores are better, indicating that the distribution of synthetic image features is closer to the distribution of real image features. A score of 0 would imply identical distributions.
• FID captures both fidelity (are the images realistic?) and diversity (does the generator produce varied outputs similar to the real dataset?). Poor fidelity or mode collapse (low diversity) will typically increase the distance between the distributions, resulting in a higher FID.

Considerations:

• Sample Size: FID requires a sufficient number of images (typically thousands) from both real and synthetic sets to reliably estimate the mean and covariance. Recommendations often suggest 10,000 or more, with at least 50,000 being preferred for benchmark comparisons.
• Pre-trained Model: The score depends on the specific features extracted by the Inception v3 model. While standard, this means FID measures similarity with respect to features important for ImageNet classification.
• Implementation: Minor differences in image preprocessing or the specific implementation of the matrix square root can lead to slight variations in scores. Using standardized implementations is recommended.

Inception Score (IS)

The Inception Score (IS) was an earlier popular metric primarily used for GANs. Unlike FID, it evaluates the generated images without direct comparison to a real dataset during the scoring phase (though the Inception model itself was trained on real data).

Intuition: The IS attempts to capture two desirable properties of generated images:

1. Quality/Fidelity: Images should contain clearly identifiable objects. When fed into the Inception classifier, the conditional probability distribution $p(y|x)$ (probability of labels $y$ given image $x$) should have low entropy – meaning the classifier is confident about what object is in the image.
2. Diversity: The generator should produce varied images spanning different classes. The marginal probability distribution $p(y) = \int p(y|x)p_g(x) dx$ (the overall distribution of labels across all generated images $x \sim p_g$) should have high entropy – meaning the images cover many different classes fairly evenly.

How it Works:

1. Classification: A set of generated images is passed through the pre-trained Inception v3 network to obtain the class probability vectors $p(y|x)$ for each image $x$.
2. Marginal Distribution: The marginal distribution $p(y)$ is estimated by averaging the probability vectors $p(y|x)$ over all generated samples.
3. KL Divergence: For each image $x$, the Kullback-Leibler (KL) divergence between its conditional distribution $p(y|x)$ and the marginal distribution $p(y)$ is calculated: $D_{KL}(p(y|x) || p(y))$. This measures how much $p(y|x)$ differs from the average distribution $p(y)$. If an image is clearly classified (low entropy $p(y|x)$) and the overall set is diverse (high entropy $p(y)$), the KL divergence will be large.
4. Averaging and Exponentiation: The final score is obtained by averaging the KL divergences over all generated images and then exponentiating the result.

The formula is:

$$IS(G) = \exp(\mathbb{E}_{x \sim p_g} [D_{KL}(p(y|x) || p(y))])$$

Interpretation:

• Higher IS scores are better. A higher score suggests the generated images are individually distinct (sharp, clear objects according to Inception) and collectively diverse (cover multiple classes).

Considerations:

• No Real Data Comparison: IS doesn't directly compare the generated distribution to the real data distribution. A generator could achieve a high IS by perfectly generating one image for each of the 1000 ImageNet classes, even if these images look nothing like the target real dataset.
• Sensitivity: IS is sensitive to the specific Inception model used.
• Known Limitations: It has been shown that IS doesn't reliably penalize intra-class mode collapse (e.g., generating only one type of dog within the "dog" class) and can be "gamed" by adversarial examples or simple generators.
• Superseded by FID: While historically significant, FID is generally preferred today as it provides a better comparison against the real data distribution.

Precision and Recall for Distributions

Inspired by precision and recall metrics in classification, these metrics were adapted to evaluate generative models by Sajjadi et al. (2018) and later refined by Kynkäänniemi et al. (2019). They provide a detailed view of fidelity and diversity by assessing the support of the real and synthetic distributions in a feature space.

Intuition:

• Precision: Measures the fraction of generated images that fall within the manifold (distribution region) of the real data. High precision means most generated images are realistic (high fidelity).
• Recall: Measures the fraction of the real data manifold that is covered by the generated images. High recall means the generator captures the variety present in the real data (high diversity).

How it Works (Simplified):

1. Feature Extraction: Similar to FID, features are extracted from both real ($X$) and synthetic ($G$) images using a pre-trained network (e.g., VGG-16 or Inception). Let these feature sets be $\phi_x$ and $\phi_g$.
2. Manifold Estimation: The challenge lies in estimating the "support" or manifold of the true data distribution based on the finite sample of real features $\phi_x$. Similarly, the support of the generated distribution is estimated from $\phi_g$. This is often done using nearest-neighbor calculations.
3. Calculating Precision: For each synthetic feature $\phi \in \phi_g$, determine if it lies "within" the estimated manifold of the real data. Precision is the fraction of synthetic features for which this is true. A common way is to check if the k-nearest neighbor of $\phi$ in the combined feature space $(\phi_x \cup \phi_g)$ belongs to the real set $\phi_x$.
4. Calculating Recall: For each real feature $\phi' \in \phi_x$, determine if it is "covered" by the estimated manifold of the synthetic data. Recall is the fraction of real features for which this is true. Similarly, this can be done by checking if the k-nearest neighbor of $\phi'$ in the combined feature space belongs to the synthetic set $\phi_g$.

Interpretation:

• High Precision: Indicates good fidelity. The generated images are largely indistinguishable from real images in the chosen feature space. Low precision suggests the generator produces unrealistic artifacts or out-of-distribution samples.
• High Recall: Indicates good diversity. The generator covers most of the variations present in the real dataset. Low recall suggests mode collapse, where the generator only produces a limited subset of the real data distribution.

Visualization: Imagine the distributions as regions in feature space. Precision asks "How much of the synthetic region overlaps with the real region?". Recall asks "How much of the real region is covered by the synthetic region?".

Diagram illustrating the concepts of Precision and Recall in feature space. Blue points represent real data features, yellow points represent synthetic data features. Precision relates to how many yellow points fall within the blue region's influence. Recall relates to how much of the blue region is covered by the yellow region's influence.

Considerations:

• Feature Space: The choice of feature extractor significantly impacts the results.
• k-NN Parameter: The number of neighbors ($k$) used in manifold estimation affects the sensitivity of the metrics.
• Computational Cost: Calculating pairwise distances or nearest neighbors can be computationally expensive for large datasets.

Choosing Image Evaluation Metrics

No single metric perfectly captures all aspects of image quality.

• FID remains a strong standard for comparing overall similarity (fidelity and diversity combined) between generated and real distributions. It correlates reasonably well with human judgment.
• IS is less favored now due to its limitations but might appear in older literature or specific contexts. Be aware of its challenges.
• Precision and Recall offer a valuable decomposition, separating the assessment of fidelity from diversity. This can be particularly insightful for diagnosing generator problems (e.g., identifying mode collapse through low recall, even if precision is high).

For a thorough evaluation, it's often best practice to report FID along with Precision and Recall calculated using a well-defined feature space and methodology. This provides a more comprehensive picture of the synthetic image generation quality. The next sections will cover metrics for other data types like text and time-series, which face their own unique evaluation challenges.