ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Advanced Diffusion Model Architectures and Training
Chapter 1: Foundation Review and Advanced Noise Schedules
Recap: Denoising Diffusion Probabilistic Models (DDPM)
Recap: Denoising Diffusion Implicit Models (DDIM)
Mathematical Underpinnings: Score Matching and ODEs
Limitations of Standard Noise Schedules
Designing Custom Noise Schedules
Learned Variance Schedules
Hands-on Practical: Implementing Noise Schedule Variants
Chapter 2: Advanced U-Net Architectures
The Standard U-Net in Diffusion Models
Attention Mechanisms in U-Nets (Self-Attention, Cross-Attention)
Integrating Time Embeddings in U-Nets
Advanced Conditioning Input Integration
Architectural Variants for Efficiency (Depth, Width, Pooling)
Normalization Techniques (GroupNorm, AdaLN)
Hands-on Practical: Modifying a U-Net with Attention
Chapter 3: Transformer-Based Diffusion Models
Motivation for Transformers in Generative Modeling
Adapting Transformers for Image Data (ViT, Patch Embeddings)
Diffusion Transformers (DiT): Architecture Overview
Conditioning in Diffusion Transformers
Comparison: U-Nets vs. Transformers for Diffusion
Implementation Considerations for DiTs
Hands-on Practical: Building a Simple DiT Block
Chapter 4: Advanced Training Techniques
Classifier Guidance: Principles and Implementation
Classifier-Free Guidance (CFG): Theory and Benefits
Implementing and Tuning CFG Scale
Advanced Loss Function Formulations (v-prediction, L_simple)
Model Parameterization (epsilon-prediction vs. x0-prediction)
Techniques for Training Stability (Gradient Clipping, EMA)
Mixed-Precision Training for Diffusion Models
Hands-on Practical: Implementing Classifier-Free Guidance
Chapter 5: Consistency Models
Motivation: The Need for Faster Sampling
Core Idea: Consistency Property
Consistency Model Training: Distillation Approach
Consistency Model Training: Standalone Approach
Sampling from Consistency Models (Single-step and Multi-step)
Architecture Considerations for Consistency Models
Trade-offs: Speed vs. Quality
Hands-on Practical: Basic Consistency Distillation
Chapter 6: Advanced Sampling and Optimization
Higher-Order Solvers (DPM-Solver, UniPC)
Stochastic Sampling Variants
Guided Sampling Refinements
Troubleshooting Sampling Issues (Artifacts, Blurriness)
Model Distillation for Diffusion
Quantization of Diffusion Models
Hardware Acceleration Considerations (GPU Kernels, Compilation)
Hands-on Practical: Comparing Advanced Samplers

Hands-on Practical: Modifying a U-Net with Attention

Here, a standard U-Net block is modified to include attention mechanisms. Adding attention, particularly self-attention, allows the model to capture long-range dependencies within image features, which can be beneficial for generating coherent structures. The focus is on adding a spatial self-attention layer within a residual block commonly found in U-Net architectures used for diffusion.

We assume you have a working understanding of PyTorch and the basic components of a U-Net (convolutional layers, normalization, activation functions, residual connections). Our goal is to augment an existing block structure.

Prerequisites: A Basic Residual Block

Let's start with a typical residual block often used in U-Nets. It might look something like this (simplified):

import torch
import torch.nn as nn
import torch.nn.functional as F

class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, time_emb_dim=None):
        super().__init__()
        self.norm1 = nn.GroupNorm(32, in_channels)
        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1)

        # Optional time embedding projection
        self.time_mlp = None
        if time_emb_dim is not None:
            self.time_mlp = nn.Sequential(
                nn.SiLU(),
                nn.Linear(time_emb_dim, out_channels * 2) # For scale and shift
            )

        self.norm2 = nn.GroupNorm(32, out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1)
        self.activation = nn.SiLU() # Swish activation is common

        # Skip connection matching
        if in_channels != out_channels:
            self.skip_connection = nn.Conv2d(in_channels, out_channels, kernel_size=1)
        else:
            self.skip_connection = nn.Identity()

    def forward(self, x, t_emb=None):
        residual = x

        h = self.norm1(x)
        h = self.activation(h)
        h = self.conv1(h)

        if self.time_mlp is not None and t_emb is not None:
            time_encoding = self.time_mlp(t_emb)
            time_encoding = time_encoding.view(h.shape[0], h.shape[1] * 2, 1, 1)
            scale, shift = torch.chunk(time_encoding, 2, dim=1)
            h = self.norm2(h) * (1 + scale) + shift # Modulate features
        else:
             h = self.norm2(h) # Apply norm if no time embedding modulation

        h = self.activation(h)
        h = self.conv2(h)

        return h + self.skip_connection(residual)

This ResidualBlock includes Group Normalization, convolutional layers, an activation function (SiLU), and handles residual connections, along with optional time embedding modulation.

Implementing a Multi-Head Self-Attention Block

Now, let's define a self-attention block suitable for image features. Since standard multi-head self-attention (MHSA) operates on sequences, we need to reshape the 2D spatial feature maps into sequences.

import math

class AttentionBlock(nn.Module):
    def __init__(self, channels, num_heads=8):
        super().__init__()
        assert channels % num_heads == 0, f"Channels ({channels}) must be divisible by num_heads ({num_heads})"
        self.num_heads = num_heads
        self.head_dim = channels // num_heads
        self.scale = 1 / math.sqrt(self.head_dim)

        self.norm = nn.GroupNorm(32, channels)
        # Query, Value projections
        self.to_qkv = nn.Conv2d(channels, channels * 3, kernel_size=1)
        # Output projection
        self.to_out = nn.Conv2d(channels, channels, kernel_size=1)

    def forward(self, x):
        b, c, h, w = x.shape
        residual = x

        qkv = self.to_qkv(self.norm(x))
        # Reshape for attention: (b, c, h, w) -> (b, c, h*w) -> (b, num_heads, head_dim, h*w) -> (b*num_heads, head_dim, h*w)
        q, k, v = qkv.chunk(3, dim=1)

        q = q.reshape(b * self.num_heads, self.head_dim, h * w)
        k = k.reshape(b * self.num_heads, self.head_dim, h * w)
        v = v.reshape(b * self.num_heads, self.head_dim, h * w)

        # Scaled dot-product attention
        # (b*num_heads, head_dim, h*w) @ (b*num_heads, h*w, head_dim) -> (b*num_heads, h*w, h*w)
        attention_scores = torch.bmm(q.transpose(1, 2), k) * self.scale
        attention_probs = F.softmax(attention_scores, dim=-1)

        # (b*num_heads, h*w, h*w) @ (b*num_heads, head_dim, h*w)' -> (b*num_heads, h*w, head_dim)
        out = torch.bmm(attention_probs, v.transpose(1, 2))

        # Reshape back: (b*num_heads, h*w, head_dim) -> (b, num_heads, h*w, head_dim) -> (b, c, h, w)
        out = out.transpose(1, 2).reshape(b, c, h, w)

        out = self.to_out(out)

        return out + residual # Add residual connection

Important points in AttentionBlock:

  1. We use GroupNorm before attention, a common practice.
  2. to_qkv projects the input into Query, Key, and Value using a 1x1 convolution.
  3. The spatial dimensions (h, w) are flattened into a sequence length h*w.
  4. Standard scaled dot-product attention is performed.
  5. The output is reshaped back to (b, c, h, w).
  6. A final 1x1 convolution projects the features back.
  7. A residual connection is added.

Integrating Attention into the U-Net

Attention blocks are typically inserted after residual blocks, especially at lower spatial resolutions (deeper in the U-Net) where feature maps are smaller (e.g., 16x16 or 8x8) and computational cost is more manageable.

Here's how you might modify the U-Net structure, specifically within the encoder or decoder loops:

# Example snippet within a U-Net encoder loop

# ... previous layers ...
x = ResidualBlock(in_channels, out_channels, time_emb_dim)(x, t_emb)

# Add attention block if dimensions are suitable (e.g., lower resolution)
if x.shape[-1] <= 16: # Apply attention at resolutions 16x16 or lower
    x = AttentionBlock(out_channels, num_heads=8)(x)

# ... potentially more residual blocks or pooling ...

Similarly, in the decoder:

# Example snippet within a U-Net decoder loop

# ... upsampling and concatenation ...
x = ResidualBlock(in_channels, out_channels, time_emb_dim)(x, t_emb)

# Add attention block consistent with encoder
if x.shape[-1] <= 16:
    x = AttentionBlock(out_channels, num_heads=8)(x)

# ... more layers ...

The diagram below illustrates where an attention block might fit within a sequence of operations in a U-Net layer.

Diagram showing the flow through a standard ResidualBlock, potentially followed by an AttentionBlock before producing the final output for that layer.

Experimentation and Approaches

Now that you have the building blocks, here are some key points:

  1. Placement: Experiment with placing attention blocks at different resolutions. Adding them too early (high resolution) can be computationally expensive. Adding them only at the bottleneck might limit their impact. Common choices include adding them after each residual block in the lower-resolution layers.
  2. Number of Heads: The num_heads parameter controls the granularity of the attention mechanism. More heads allow attending to different feature subspaces but increase computation slightly. Values like 4, 8, or 16 are common. Ensure the channel dimension is divisible by num_heads.
  3. Cross-Attention: For conditional models (e.g., text-to-image), you would implement a CrossAttentionBlock instead of or in addition to self-attention. The key and value inputs would come from the conditioning context (e.g., projected text embeddings), while the query would come from the image features x. This is typically added in similar locations to self-attention.
  4. Computational Cost: Attention mechanisms, especially self-attention, have a computational cost quadratic in the sequence length (which is h*w here). This is why they are often restricted to lower resolutions.
  5. Training Stability: Ensure appropriate normalization (like GroupNorm used here) and residual connections are used around attention blocks to maintain stable training.

By integrating attention mechanisms, you equip your U-Net to better model complex spatial relationships and potentially integrate conditioning information more effectively, pushing the capabilities of your diffusion model. Remember to test these modifications carefully, monitoring both sample quality and training performance.

© 2026 ApX Machine Learning