Advanced Diffusion Model Architectures and Training
Chapter 1: Foundation Review and Advanced Noise Schedules
Recap: Denoising Diffusion Probabilistic Models (DDPM)
Recap: Denoising Diffusion Implicit Models (DDIM)
Mathematical Underpinnings: Score Matching and ODEs
Limitations of Standard Noise Schedules
Designing Custom Noise Schedules
Learned Variance Schedules
Hands-on Practical: Implementing Noise Schedule Variants
Chapter 2: Advanced U-Net Architectures
The Standard U-Net in Diffusion Models
Attention Mechanisms in U-Nets (Self-Attention, Cross-Attention)
Integrating Time Embeddings in U-Nets
Advanced Conditioning Input Integration
Architectural Variants for Efficiency (Depth, Width, Pooling)
Normalization Techniques (GroupNorm, AdaLN)
Hands-on Practical: Modifying a U-Net with Attention
Chapter 3: Transformer-Based Diffusion Models
Motivation for Transformers in Generative Modeling
Adapting Transformers for Image Data (ViT, Patch Embeddings)
Diffusion Transformers (DiT): Architecture Overview
Conditioning in Diffusion Transformers
Comparison: U-Nets vs. Transformers for Diffusion
Implementation Considerations for DiTs
Hands-on Practical: Building a Simple DiT Block
Chapter 4: Advanced Training Techniques
Classifier Guidance: Principles and Implementation
Classifier-Free Guidance (CFG): Theory and Benefits
Implementing and Tuning CFG Scale
Advanced Loss Function Formulations (v-prediction, L_simple)
Model Parameterization (epsilon-prediction vs. x0-prediction)
Techniques for Training Stability (Gradient Clipping, EMA)
Mixed-Precision Training for Diffusion Models
Hands-on Practical: Implementing Classifier-Free Guidance
Chapter 5: Consistency Models
Motivation: The Need for Faster Sampling
Core Idea: Consistency Property
Consistency Model Training: Distillation Approach
Consistency Model Training: Standalone Approach
Sampling from Consistency Models (Single-step and Multi-step)
Architecture Considerations for Consistency Models
Trade-offs: Speed vs. Quality
Hands-on Practical: Basic Consistency Distillation
Chapter 6: Advanced Sampling and Optimization
Higher-Order Solvers (DPM-Solver, UniPC)
Stochastic Sampling Variants
Guided Sampling Refinements
Troubleshooting Sampling Issues (Artifacts, Blurriness)
Model Distillation for Diffusion
Quantization of Diffusion Models
Hardware Acceleration Considerations (GPU Kernels, Compilation)
Hands-on Practical: Comparing Advanced Samplers
Classifier Guidance: Principles and Implementation
Was this section helpful?
- Diffusion Models Beat GANs on Image Synthesis, Prafulla Dhariwal and Alex Nichol, 2021 Advances in Neural Information Processing Systems (NeurIPS 2021) DOI: 10.48550/arXiv.2105.05233 - This foundational paper introduces classifier guidance, demonstrating how to steer diffusion models toward specific classes using gradients from a pre-trained classifier. It shows that guided diffusion models can achieve high quality in image synthesis.
- Denoising Diffusion Probabilistic Models, Jonathan Ho, Ajay Jain, and Pieter Abbeel, 2020 Advances in Neural Information Processing Systems (NeurIPS 2020), Vol. 33 DOI: 10.48550/arXiv.2006.11239 - This paper introduces the Denoising Diffusion Probabilistic Model (DDPM) framework, which classifier guidance modifies. It describes the forward and reverse diffusion processes, forming the basis for many modern diffusion models.
- Generative Modeling by Estimating Gradients of the Data Distribution, Yang Song, Stefano Ermon, 2019 Advances in Neural Information Processing Systems (NeurIPS 2019), Vol. 32 DOI: 10.48550/arXiv.1907.05600 - This paper introduces score-based generative models, establishing a theoretical framework for using gradients of the log-density (score functions) for generative tasks. This underpins the gradient-based steering mechanism used in classifier guidance.
- Improved Denoising Diffusion Probabilistic Models, Alex Nichol, Prafulla Dhariwal, 2021 International Conference on Machine Learning (ICML 2021) DOI: 10.48550/arXiv.2102.09672 - This work builds upon DDPMs and includes further enhancements and empirical studies on techniques like classifier guidance. It offers additional insights into improving the performance and sample quality of diffusion models.