StackGAN: Text-to-Image Synthesis

Generating images that correspond to detailed text descriptions represents a significant step towards more intuitive human-computer interaction and creative content generation. While basic conditional GANs (cGANs) can generate images based on simple labels (like classes), generating high-resolution, photorealistic images from complex free-form text descriptions presents a much greater challenge. The text provides semantic information but lacks the pixel-level detail required for a high-fidelity image. Generating everything in one go often leads to blurry results or images that only vaguely match the description.

StackGAN addresses this challenge with a hierarchical approach, breaking down the complex text-to-image problem into more manageable sub-problems handled by stacked generative networks.

The Two-Stage Architecture

The core idea behind StackGAN is to use a two-stage generative process:

Stage-I GAN: This network focuses on sketching a preliminary, low-resolution version of the image. It takes a text embedding (derived from the input description using a pre-trained text encoder like a character-level CNN-RNN or BERT) and a random noise vector $z$ as input. Its goal is to generate an image that captures the rough layout, colors, and basic shapes described in the text, ignoring fine-grained details and focusing on semantic coherence. The output is typically a low-resolution image (e.g., 64x64 pixels).
Stage-II GAN: This network takes the low-resolution image generated by Stage-I and the original text embedding as input. Its objective is to refine the initial sketch, correct defects, add details, and significantly increase the image resolution (e.g., to 256x256 pixels). By conditioning on both the Stage-I image and the text embedding, Stage-II can focus on adding realism and detail consistent with the initial structure and the textual description.

This staged approach allows the model to first focus on the global structure and semantics (Stage-I) before concentrating on pixel-level refinement and detail (Stage-II).

StackGAN architecture overview. Text is encoded, augmented, and used with noise $z$ in Stage-I to produce a low-resolution sketch. Stage-II takes this sketch and the text embedding to generate a refined, high-resolution image. Both stages involve adversarial training.

Conditioning Augmentation

A potential issue in conditional generation is that the space of text embeddings might be sparse or not sufficiently smooth. If the embeddings for similar descriptions are far apart, the generator might struggle to generalize. StackGAN introduces Conditioning Augmentation (CA) to address this.

Instead of directly using the text embedding $\phi_t$ produced by the encoder, CA models the embedding as a Gaussian distribution $N(\mu(\phi_t), \Sigma(\phi_t))$ , where the mean $\mu(\phi_t)$ and diagonal covariance matrix $\Sigma(\phi_t)$ are functions of the original embedding $\phi_t$ (often implemented using small fully connected layers). During training, a conditioning variable $c$ is sampled from this distribution:

\hat{c} \sim N(\mu(\phi_t), \Sigma(\phi_t))

This sampled $\hat{c}$ is then used as the condition for the Stage-I generator $G_1(z, \hat{c})$ . This sampling process encourages robustness to small variations in the text embedding and creates a smoother conditioning manifold, making the generator's learning task easier. A regularization term, the Kullback-Leibler (KL) divergence between the learned distribution $N(\mu(\phi_t), \Sigma(\phi_t))$ and a standard Gaussian $N(0, I)$ , is added to the generator's objective to prevent the variance from collapsing:

L_{CA} = D_{KL}(N(\mu(\phi_t), \Sigma(\phi_t)) || N(0, I))

The Stage-II GAN typically uses the original text embedding $\phi_t$ directly for conditioning, as it primarily focuses on adding detail based on the already established structure from Stage-I and the semantic guidance from the text.

Training and Loss Functions

Training StackGAN involves training Stage-I first, and then training Stage-II using the outputs of the trained Stage-I generator.

Stage-I: The generator $G_1$ and discriminator $D_1$ are trained adversarially. $G_1$ tries to generate realistic low-resolution images matching the sampled condition $\hat{c}$ , incorporating the $L_{CA}$ regularization. $D_1$ tries to distinguish between real low-resolution images (downsampled from the dataset) paired with their corresponding text condition $\hat{c}$ , and fake low-resolution images from $G_1$ paired with $\hat{c}$ .
Stage-II: The generator $G_2$ and discriminator $D_2$ are trained adversarially. $G_2$ takes the low-resolution output $s_0 = G_1(z, \hat{c})$ and the original text embedding $\phi_t$ as input, aiming to produce a realistic high-resolution image $s_1 = G_2(s_0, \phi_t)$ . The discriminator $D_2$ learns to distinguish real high-resolution images paired with their text embedding $\phi_t$ from the fake high-resolution images $s_1$ generated by $G_2$ , also conditioned on $\phi_t$ .

Both stages use standard GAN loss functions (or variants like Wasserstein loss) combined with the conditioning information. The point is that Stage-II's discriminator must evaluate not only the realism of the high-resolution image but also its consistency with the text embedding $\phi_t$ .

StackGAN demonstrated a significant improvement in generating high-resolution images (e.g., 256x256) from text descriptions compared to previous single-stage approaches. The hierarchical refinement strategy proved effective for tackling the complexity of mapping abstract text to detailed pixels. Subsequent work, like StackGAN++, further refined this approach with improved loss functions and training stability techniques.

Was this section helpful?

References

StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks, Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris N. Metaxas, 2017 2017 IEEE International Conference on Computer Vision (IEEE) DOI: 10.1109/ICCV.2017.394 - Presents the original two-stage StackGAN architecture for generating high-resolution images from text descriptions, introducing conditioning augmentation.
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks, Han Zhang, Tao Xu, Hongxu Ma, Zhiming Liu, Gang Huang, Shaoting Zhang, and Dimitris Metaxas, 2018 International Conference on Learning Representations (ICLR) (OpenReview.net) DOI: 10.48550/arXiv.1710.05465 - Extends the original StackGAN by improving training stability, loss functions, and generator architecture for enhanced text-to-image synthesis.
Conditional Generative Adversarial Nets, Mehdi Mirza and Simon Osindero, 2014 arXiv preprint arXiv:1411.1784 DOI: 10.48550/arXiv.1411.1784 - Introduces the concept of conditional GANs, allowing generation of images based on specific conditions like class labels, a precursor to text-to-image generation.
Generative Adversarial Nets, Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, Yoshua Bengio, 2014 Advances in Neural Information Processing Systems, Vol. 27 (NeurIPS) DOI: 10.48550/arXiv.1406.2661 - This paper introduces Generative Adversarial Networks, a core component of StackGAN and other advanced generation models.