While U-Net architectures based on convolutional neural networks (CNNs) have been the standard backbone for many successful diffusion models, the transformer architecture, initially dominant in natural language processing, has shown significant promise for generative tasks, including image synthesis. Transformers excel at modeling long-range dependencies, offering a different approach to capturing relationships within data compared to the locality bias inherent in CNNs.

This chapter examines how transformer architectures can be effectively utilized within the diffusion model framework. We will cover:

• The motivation for using transformers in generative modeling and how they are adapted for image data using techniques like patch embeddings, drawing from concepts like the Vision Transformer (ViT).
• The specific architecture of Diffusion Transformers (DiTs), which replace the U-Net backbone with transformer blocks.
• Methods for incorporating conditioning information, such as class labels or text embeddings, into DiT models.
• A comparison of the trade-offs between U-Net and transformer backbones regarding performance, scalability, and computational requirements.
• Practical considerations for implementing and training transformer-based diffusion models.

By the end of this chapter, you will understand the structure and function of transformer-based diffusion models and be prepared to analyze and implement them.