Advanced Diffusion Model Architectures and Training
Chapter 1: Foundation Review and Advanced Noise Schedules
Recap: Denoising Diffusion Probabilistic Models (DDPM)
Recap: Denoising Diffusion Implicit Models (DDIM)
Mathematical Underpinnings: Score Matching and ODEs
Limitations of Standard Noise Schedules
Designing Custom Noise Schedules
Learned Variance Schedules
Hands-on Practical: Implementing Noise Schedule Variants
Chapter 2: Advanced U-Net Architectures
The Standard U-Net in Diffusion Models
Attention Mechanisms in U-Nets (Self-Attention, Cross-Attention)
Integrating Time Embeddings in U-Nets
Advanced Conditioning Input Integration
Architectural Variants for Efficiency (Depth, Width, Pooling)
Normalization Techniques (GroupNorm, AdaLN)
Hands-on Practical: Modifying a U-Net with Attention
Chapter 3: Transformer-Based Diffusion Models
Motivation for Transformers in Generative Modeling
Adapting Transformers for Image Data (ViT, Patch Embeddings)
Diffusion Transformers (DiT): Architecture Overview
Conditioning in Diffusion Transformers
Comparison: U-Nets vs. Transformers for Diffusion
Implementation Considerations for DiTs
Hands-on Practical: Building a Simple DiT Block
Chapter 4: Advanced Training Techniques
Classifier Guidance: Principles and Implementation
Classifier-Free Guidance (CFG): Theory and Benefits
Implementing and Tuning CFG Scale
Advanced Loss Function Formulations (v-prediction, L_simple)
Model Parameterization (epsilon-prediction vs. x0-prediction)
Techniques for Training Stability (Gradient Clipping, EMA)
Mixed-Precision Training for Diffusion Models
Hands-on Practical: Implementing Classifier-Free Guidance
Chapter 5: Consistency Models
Motivation: The Need for Faster Sampling
Core Idea: Consistency Property
Consistency Model Training: Distillation Approach
Consistency Model Training: Standalone Approach
Sampling from Consistency Models (Single-step and Multi-step)
Architecture Considerations for Consistency Models
Trade-offs: Speed vs. Quality
Hands-on Practical: Basic Consistency Distillation
Chapter 6: Advanced Sampling and Optimization
Higher-Order Solvers (DPM-Solver, UniPC)
Stochastic Sampling Variants
Guided Sampling Refinements
Troubleshooting Sampling Issues (Artifacts, Blurriness)
Model Distillation for Diffusion
Quantization of Diffusion Models
Hardware Acceleration Considerations (GPU Kernels, Compilation)
Hands-on Practical: Comparing Advanced Samplers
Designing Custom Noise Schedules
Was this section helpful?
- Denoising Diffusion Probabilistic Models, Jonathan Ho, Ajay N. Jain, Pieter Abbeel, 2020 Advances in Neural Information Processing Systems (NeurIPS), Vol. 33 (Curran Associates, Inc.) DOI: 10.55919/neurips-2020-00101 - This foundational paper introduces Denoising Diffusion Probabilistic Models and their forward/reverse processes, including the original linear noise schedule. It is important for understanding the basics of noise schedules.
- Improved Denoising Diffusion Probabilistic Models, Alexander Quinn Nichol, Prafulla Dhariwal, 2021 Proceedings of the 38th International Conference on Machine Learning, Vol. 139 (PMLR) DOI: 10.1109/ICCV48922.2021.00971 - This paper introduced the cosine noise schedule as an enhancement over the linear schedule, directly addressing the need for exploring different noise schedule designs.
- Score-Based Generative Modeling through Stochastic Differential Equations, Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole, 2021 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.2011.13456 - This paper presents a unified framework for score-based generative models using stochastic differential equations, offering a continuous-time view of noise schedules and their link to the signal-to-noise ratio (SNR), a concept for principled schedule design.
- Elucidating the Design Space of Diffusion-Based Generative Models, Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, Timo Aila, 2022 Advances in Neural Information Processing Systems, Vol. 35 DOI: 10.55919/neurips-2022-ed1209b0 - This paper systematically analyzes various design options for diffusion models, including the specific parameterization and design of noise schedules (sigma schedules), and their effect on sample quality and training efficiency. It gives specific guidance on creating effective schedules.