Inception Score (IS): Formulation and Limitations

While visual inspection offers valuable qualitative observations, it's subjective and difficult to scale. Automated, quantitative metrics are needed to thoroughly assess GAN performance, compare different models, and track training progress. The Inception Score (IS) is one of the earliest and most widely adopted metrics. It aims to capture two desirable properties of generated images:

1. Image Quality: Generated images should look like something specific and recognizable. When fed into a pre-trained image classifier (like Inception-v3 trained on ImageNet), the classifier should assign high probability to a single class for each image.
2. Image Diversity: The generator should produce a wide variety of images covering different types or classes present in the real data. Across a large set of generated images, the distribution of predicted classes should be diverse or relatively uniform.

Formulation of the Inception Score

The Inception Score uses a pre-trained Inception-v3 network, chosen for its strong performance on the ImageNet dataset. The core idea is to measure the properties outlined above using the classifier's predictions on generated samples.

Let $x$ be an image generated by the generator $G$, so $x \sim p_g$. We pass $x$ through the Inception-v3 model to obtain the conditional probability distribution $p(y|x)$, where $y$ represents the class label (out of the 1000 ImageNet classes).

1. Quality Measurement: If image $x$ is high-quality and clearly depicts an object from one of the ImageNet classes, the distribution $p(y|x)$ should have low entropy. That is, the model should be confident about which class $x$ belongs to. Low entropy means the probability mass is concentrated on a few classes (ideally one).

2. Diversity Measurement: If the generator produces diverse images spanning many classes, the marginal distribution $p(y)$ over all generated images should have high entropy. This marginal distribution is obtained by averaging the conditional distributions over all generated samples:

   $$p(y) = \int_x p(y|x) p_g(x) dx$$

   In practice, $p(y)$ is estimated by averaging $p(y|x)$ over a large batch of generated samples $\{x_i\}_{i=1}^N$:

   $$\hat{p}(y) \approx \frac{1}{N} \sum_{i=1}^N p(y|x_i)$$

   High entropy for $\hat{p}(y)$ indicates that the generated images, as classified by Inception-v3, cover a wide range of classes relatively uniformly.

The Inception Score combines these two ideas using the Kullback-Leibler (KL) divergence. Specifically, it measures the divergence between the conditional distribution $p(y|x)$ for each image and the marginal distribution $p(y)$. We want $p(y|x)$ to be peaky (low entropy) and $p(y)$ to be uniform (high entropy). The KL divergence $D_{KL}(p(y|x) || p(y))$ quantifies how much $p(y|x)$ differs from $p(y)$. A large KL divergence here is desirable, indicating that individual images correspond strongly to specific classes, while the overall class usage is diverse.

The final Inception Score is calculated by taking the average KL divergence over all generated samples and exponentiating the result:

$$IS(G) = \exp\left( \mathbb{E}_{x \sim p_g} [ D_{KL}(p(y|x) || p(y)) ] \right)$$

In practice, this is approximated using a large sample set:

$$IS(G) \approx \exp\left( \frac{1}{N} \sum_{i=1}^N D_{KL}(p(y|x_i) || \hat{p}(y)) \right)$$

A higher Inception Score is generally interpreted as better performance, suggesting the generator produces images that are both high-quality (easily classifiable) and diverse (covering many classes).

Limitations of the Inception Score

Despite its intuitive appeal and widespread use, the Inception Score has several significant limitations that are important to understand, especially when working with advanced GANs:

1. Dependence on the Pre-trained Model: IS is intrinsically tied to the Inception-v3 model trained on ImageNet. It measures features that this specific classifier deems important for discriminating between ImageNet classes. These features might not perfectly align with human perception of image quality or the characteristics of your target dataset if it differs significantly from ImageNet (e.g., medical scans, abstract art, specific face datasets).

2. No Comparison to Real Data: The score is calculated solely based on generated images. It doesn't directly compare the distribution of generated images ($p_g$) to the distribution of real images ($p_{data}$). A generator could theoretically achieve a high IS by producing diverse, clearly classifiable images that bear no resemblance to the actual training data. For instance, generating perfect images of dogs and cats might yield a good IS, even if the training data consisted only of cars.

3. Sensitivity to ImageNet Classes: The score inherently rewards generators that produce images resembling the 1000 classes in ImageNet. If your GAN is trained on a dataset with different object categories, the IS might not be a meaningful measure of performance. A GAN trained to generate MNIST digits, for example, would likely receive a very low IS because digits don't strongly map to ImageNet classes like "dog" or "car".

4. Limited Ability to Detect Mode Collapse: While severe mode collapse (generating only one or very few distinct image types) should lead to a low-entropy marginal distribution $p(y)$ and thus a lower IS, the metric isn't foolproof. A generator could collapse to producing only a handful of ImageNet classes perfectly. If these few classes are diverse among themselves, the marginal entropy might still be reasonably high, masking the lack of diversity relative to the full dataset.

5. Averaging Nature: The score averages the KL divergence over samples. A generator might produce many good samples and a few terrible ones; the average score might still look acceptable, hiding potential failure modes.

6. Computational Cost: Calculating IS requires generating a large number of samples (typically tens of thousands) and performing inference using the relatively large Inception-v3 model for each sample, which can be computationally demanding.

The Inception Score provides a single number summarizing quality and diversity based on a pre-trained classifier's perspective. However, it doesn't compare generated samples to real ones and is biased towards ImageNet features.

Because of these limitations, while IS was an important step forward in GAN evaluation, it's often supplemented or replaced by newer metrics like the Fréchet Inception Distance (FID), which directly compares statistics of generated samples to real samples using features from the same Inception network. Understanding IS provides valuable context for appreciating the evolution of GAN evaluation techniques.