Higher-Order Solvers (DPM-Solver, UniPC)

While DDIM provides a significant speedup over DDPM by utilizing a deterministic ODE approximation, it still requires a relatively large number of steps (typically 50-200) to achieve high-fidelity results. This is because DDIM is essentially a first-order numerical solver (akin to the Euler method) for the underlying probability flow ODE that governs the reverse diffusion process. The reverse process is described by an ordinary differential equation (ODE):

$$dx = f(x_t, t) dt$$

where $f(x_t, t)$ depends on the score function $\nabla_{x_t} \log p_t(x_t)$ or the noise prediction $\epsilon_\theta(x_t, t)$. First-order methods like Euler (and by extension, DDIM) approximate the solution by taking small steps based only on the current state's derivative. Achieving high accuracy with such methods necessitates many small steps, leading to slower inference.

To overcome this limitation, we can employ more sophisticated numerical methods designed for solving ODEs. These higher-order solvers utilize information from multiple previous steps or intermediate points within a step to achieve a more accurate approximation of the ODE trajectory. This increased accuracy per step allows for larger step sizes, significantly reducing the total number of function evaluations (NFE) required to generate a sample, often achieving comparable or even superior quality to DDIM in far fewer steps (e.g., 10-25).

Two prominent families of higher-order solvers adapted for diffusion models are DPM-Solver and UniPC.

DPM-Solver (Diffusion Probabilistic Model Solver)

DPM-Solver is a family of solvers specifically tailored for the structure of the diffusion ODE. It uses the insight that the diffusion ODE often has a semi-linear form, allowing for efficient and accurate solutions using techniques like exponential integrators.

The core idea is to approximate the ODE solution more accurately over a larger interval $\Delta t$. DPM-Solver comes in different orders (e.g., DPM-Solver-2, DPM-Solver-3), where higher orders use more information for potentially better accuracy, although sometimes at the cost of stability.

A particularly effective variant is DPM-Solver++. It often combines the exponential integrator approach with a data prediction term (predicting $x_0$ and using that to guide the step), leading to very stable and high-quality results even with very few steps (often < 20).

Advantages:

• Speed: Significantly reduces the number of steps needed compared to DDIM for similar quality.
• Quality: Can often achieve high fidelity generation in 15-25 steps.
• Specialized: Designed specifically for the structure of diffusion ODEs.

Key Points:

• The optimal order and specific variant (e.g., DPM-Solver++ SDE) might depend on the specific model and dataset.
• While libraries abstract the complexity, understanding the basic principle helps in tuning.

UniPC (Unified Predictor-Corrector)

UniPC offers an alternative approach based on classical predictor-corrector methods for solving ODEs. It aims to unify previous methods like DDIM (predictor) and Analytic-DPM (corrector) into a single framework.

A predictor-corrector method works in two stages per step:

1. Predictor: Makes an initial guess for the solution at the next timestep (similar to DDIM or a DPM-Solver step).
2. Corrector: Refines this prediction using additional information, often involving another model evaluation or using the predicted score.

UniPC cleverly adapts this scheme for diffusion models. By applying one or more corrector steps, it can improve the accuracy of the prediction made in the first stage, effectively allowing for even larger step sizes or better quality at the same step count compared to predictor-only methods.

Advantages:

• Speed: Often achieves state-of-the-art results in terms of quality vs. number of steps, sometimes requiring only 10-15 steps.
• Flexibility: Provides a unified framework potentially adaptable to different diffusion formulations.
• Stability: Corrector steps can sometimes help stabilize the integration process.

Key Points:

• May involve slightly more computation per step if multiple corrector evaluations are used, though the reduction in total steps usually leads to a net speedup.
• Has its own set of hyperparameters to tune (e.g., variant, order).

Comparison and Practical Use

Both DPM-Solver++ and UniPC represent significant advancements over DDIM for fast sampling. They routinely enable high-quality image generation in 10-25 steps, a dramatic reduction from the 50-200 steps often needed by DDIM.

Comparison showing how higher-order solvers like DPM-Solver++ and UniPC can reach high quality (low FID score) much faster (fewer steps) than DDIM. Actual performance varies by model and task.

The "best" choice often depends on the specific diffusion model architecture, the dataset it was trained on, and the desired trade-off between speed and absolute maximum quality. DPM-Solver++ is a very strong and popular baseline, while UniPC often pushes the boundary for the minimum number of steps.

Fortunately, integrating these solvers into practical workflows is usually straightforward, thanks to libraries like Hugging Face diffusers. Switching schedulers often involves changing only a few lines of code where the scheduler is initialized.

# Example using Hugging Face Diffusers (Illustrative)
from diffusers import DiffusionPipeline
from diffusers import DDIMScheduler, DPMSolverMultistepScheduler, UniPCMultistepScheduler

model_id = "stabilityai/stable-diffusion-2-1-base" # Or your custom model

# Load pipeline (example for Stable Diffusion)
pipe = DiffusionPipeline.from_pretrained(model_id)

# --- Select your scheduler ---
# pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
# pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# --- ---

prompt = "A photograph of an astronaut riding a horse on the moon"
num_inference_steps = 20 # Use fewer steps with advanced solvers!

image = pipe(prompt, num_inference_steps=num_inference_steps).images[0]
# image.save("generated_image.png")

Experimenting with these advanced solvers is highly recommended when optimizing diffusion models for inference speed. By replacing DDIM with DPM-Solver++ or UniPC, you can often achieve substantial reductions in generation time without compromising significantly on the quality of the output. This makes complex diffusion models much more practical for real-time applications and resource-constrained environments. Remember to also experiment with the num_inference_steps parameter, as these solvers are designed to work well with much lower values than DDIM.