Evaluating Conditional Generation Models

Evaluating generative models that produce outputs based on specific conditions, such as class labels, text descriptions, or other guiding inputs, requires specialized approaches. While metrics like FID and IS assess overall sample quality and diversity, they don't directly measure whether the generated output faithfully adheres to the provided condition. Evaluating conditional models, therefore, involves assessing two primary aspects:

1. Conditional Consistency (Fidelity): Does the generated sample accurately reflect the given condition?
2. Conditional Quality and Diversity: Are the samples generated for a specific condition high-quality and diverse?

Let's examine techniques for measuring these aspects.

Assessing Conditional Consistency

The goal here is to quantify how well the generated output matches the input condition. The methods vary depending on the type of condition.

Class-Conditional Generation

For models conditioned on class labels (e.g., generating images of specific dog breeds), a common approach is to use a pre-trained classifier.

1. Train a Classifier: Train a standard image classifier on the real dataset used to train the generative model. This classifier should achieve high accuracy on a held-out test set of real images.
2. Evaluate Generated Samples: Generate synthetic samples for each class condition. Feed these generated samples, along with their intended labels, into the pre-trained classifier.
3. Measure Accuracy: Calculate the classification accuracy on the generated samples. High accuracy suggests the generator is producing images that the classifier recognizes as belonging to the target class, indicating good conditional consistency.

This is sometimes referred to as "Classification Accuracy Score" (CAS) in literature. A variation involves calculating the KL divergence between the predicted class distribution for generated samples of a target class and a one-hot vector representing the target class. Lower KL divergence indicates better alignment.

However, be mindful of potential issues:

• The classifier might be biased towards features learned from the real data, which might not perfectly align with the generator's output distribution.
• A generator might learn to produce samples that "fool" the specific classifier used for evaluation, even if the samples aren't genuinely representative of the class. Using multiple diverse classifiers can help mitigate this.

Text-to-Image Generation

For models generating images from text prompts, evaluating conditional consistency requires measuring the semantic alignment between the text and the image. The most prevalent metric for this is the CLIP Score.

CLIP (Contrastive Language–Image Pre-training) is a model trained by OpenAI on a massive dataset of image-text pairs. It learns a shared embedding space where corresponding images and text descriptions have high cosine similarity.

To calculate the CLIP score:

1. Generate Image: Create an image using the text prompt $t$ with your generative model. Let the generated image be $I$.
2. Embed Text and Image: Use the pre-trained CLIP model to obtain the text embedding $e_t$ for $t$ and the image embedding $e_I$ for $I$.
3. Calculate Cosine Similarity: The CLIP score for this single pair is the cosine similarity between the embeddings: $$\text{CLIP Score}(t, I) = \frac{e_t \cdot e_I}{\|e_t\| \|e_I\|}$$
4. Average: Typically, the CLIP score is averaged over a large set of generated text-image pairs from a benchmark dataset (like MS-COCO captions).

A higher average CLIP score indicates better alignment between the generated images and their corresponding text prompts. While widely used, CLIP itself has limitations and biases inherited from its training data, which can influence the score.

Other Conditional Inputs

For other types of conditions (e.g., generating an image based on a segmentation map, style transfer based on a reference image), evaluation often relies on domain-specific metrics.

• Segmentation Map to Image: You might measure the Intersection over Union (IoU) between the input segmentation map and a segmentation map predicted from the generated image using a pre-trained segmentation model.
• Style Transfer: Metrics might involve comparing style statistics (like Gram matrices) between the generated image and the style reference, or perceptual similarity metrics comparing the generated image's content to the original content image.

Assessing Conditional Quality and Diversity

Beyond checking if the output matches the condition, we need to ensure the quality and diversity within each condition are satisfactory. A model might produce excellent images for one class but poor ones for another, or it might suffer from mode collapse only for certain conditions.

Standard evaluation metrics like FID, Precision, Recall, and KID can be adapted for conditional assessment:

1. Per-Condition Calculation: Generate a sufficient number of samples for each distinct condition (e.g., for each class label).
2. Subset Evaluation: Calculate metrics like FID by comparing the distribution of generated samples for a specific condition (e.g., 'cat' images) against the distribution of real samples belonging only to that condition.

For instance, you would compute:

• FID(Real 'cat' images, Generated 'cat' images)
• FID(Real 'dog' images, Generated 'dog' images)
• ... and so on for all classes.

This provides a more granular view than a single global FID score. It can reveal if the model performs unevenly across different conditions or suffers from intra-class mode collapse (low diversity for a specific class).

The chart shows FID scores calculated separately for images generated under four different class conditions. Class C exhibits a significantly higher FID, indicating lower quality or diversity for generated samples belonging to that specific class compared to the others.

Similarly, precision and recall can be calculated per condition to understand if the generator covers the diversity of real samples within that condition (recall) and if the generated samples are realistic for that condition (precision).

Combining Metrics

In practice, evaluating conditional generative models often involves reporting a combination of metrics:

• A metric for conditional consistency (e.g., Classifier Accuracy for class-conditional, CLIP Score for text-to-image).
• A metric for overall quality/diversity (e.g., global FID across all conditions).
• Metrics for conditional quality/diversity (e.g., average per-class FID, or reporting the range/variance of per-class FID).

This multi-faceted approach provides a more comprehensive understanding of the model's capabilities and weaknesses, guiding further development and optimization efforts. Remember that choosing the right evaluation metrics depends heavily on the specific task and the nature of the conditions involved.