Advanced Diffusion Model Architectures and Training
Chapter 1: Foundation Review and Advanced Noise Schedules
Recap: Denoising Diffusion Probabilistic Models (DDPM)
Recap: Denoising Diffusion Implicit Models (DDIM)
Mathematical Underpinnings: Score Matching and ODEs
Limitations of Standard Noise Schedules
Designing Custom Noise Schedules
Learned Variance Schedules
Hands-on Practical: Implementing Noise Schedule Variants
Chapter 2: Advanced U-Net Architectures
The Standard U-Net in Diffusion Models
Attention Mechanisms in U-Nets (Self-Attention, Cross-Attention)
Integrating Time Embeddings in U-Nets
Advanced Conditioning Input Integration
Architectural Variants for Efficiency (Depth, Width, Pooling)
Normalization Techniques (GroupNorm, AdaLN)
Hands-on Practical: Modifying a U-Net with Attention
Chapter 3: Transformer-Based Diffusion Models
Motivation for Transformers in Generative Modeling
Adapting Transformers for Image Data (ViT, Patch Embeddings)
Diffusion Transformers (DiT): Architecture Overview
Conditioning in Diffusion Transformers
Comparison: U-Nets vs. Transformers for Diffusion
Implementation Considerations for DiTs
Hands-on Practical: Building a Simple DiT Block
Chapter 4: Advanced Training Techniques
Classifier Guidance: Principles and Implementation
Classifier-Free Guidance (CFG): Theory and Benefits
Implementing and Tuning CFG Scale
Advanced Loss Function Formulations (v-prediction, L_simple)
Model Parameterization (epsilon-prediction vs. x0-prediction)
Techniques for Training Stability (Gradient Clipping, EMA)
Mixed-Precision Training for Diffusion Models
Hands-on Practical: Implementing Classifier-Free Guidance
Chapter 5: Consistency Models
Motivation: The Need for Faster Sampling
Core Idea: Consistency Property
Consistency Model Training: Distillation Approach
Consistency Model Training: Standalone Approach
Sampling from Consistency Models (Single-step and Multi-step)
Architecture Considerations for Consistency Models
Trade-offs: Speed vs. Quality
Hands-on Practical: Basic Consistency Distillation
Chapter 6: Advanced Sampling and Optimization
Higher-Order Solvers (DPM-Solver, UniPC)
Stochastic Sampling Variants
Guided Sampling Refinements
Troubleshooting Sampling Issues (Artifacts, Blurriness)
Model Distillation for Diffusion
Quantization of Diffusion Models
Hardware Acceleration Considerations (GPU Kernels, Compilation)
Hands-on Practical: Comparing Advanced Samplers
Troubleshooting Sampling Issues (Artifacts, Blurriness)
Was this section helpful?
- DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Models, Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, Jun Zhu, 2022 NeurIPS 2022 DOI: 10.48550/arXiv.2206.00927 - This paper introduces fast ODE solvers that reduce sampling steps, relevant for understanding issues stemming from insufficient steps or aggressive solver settings.
- Score-Based Generative Modeling through Stochastic Differential Equations, Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole, 2020 ICLR 2021 DOI: 10.48550/arXiv.2011.13456 - Offers a unified theoretical framework for diffusion models using SDEs and ODEs, providing context for the properties and stability of different sampler types.
- High-Resolution Image Synthesis with Latent Diffusion Models, Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer, 2022 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) DOI: 10.48550/arXiv.2112.10752 - Introduces Latent Diffusion Models and describes the VAE's function, directly addressing VAE-related issues that cause blurriness or color shifts.
- Elucidating the Design Space of Diffusion-Based Generative Models, Tero Karras, Miika Aittala, Timo Aila, Samuli Laine, 2022 NeurIPS 2022 DOI: 10.48550/arXiv.2206.00364 - Systematically analyzes the influence of various design choices, such as noise schedules and architectures, on diffusion model quality, offering insights into common artifact causes.