Core Idea: Consistency Property

The iterative nature of standard diffusion samplers poses a practical challenge due to high computational cost. This iterative process often arises from the underlying connection between diffusion models and Ordinary Differential Equations (ODEs). Consistency models offer a distinct approach to circumvent this multi-step generation process. This approach is founded on the consistency property, which is derived from the continuous-time perspective of diffusion.

The Probability Flow ODE

Diffusion models can be interpreted through the lens of continuous stochastic processes and their corresponding ODEs. Specifically, the generation process can be viewed as reversing a forward process described by an ODE, often referred to as the Probability Flow (PF) ODE. Let $x(t)$ represent the state (e.g., an image) at time $t$ , where $t$ ranges from $t=0$ (real data) to $t=T$ (approximately pure noise). The PF ODE describes the path, or trajectory, that transforms a data point $x(0)$ into noise $x(T)$ and vice-versa. Standard diffusion samplers essentially approximate the solution to this ODE by taking many small, discrete steps backward from $x(T)$ to estimate $x(0)$ .

Defining Consistency

The central idea behind consistency models revolves around a specific property of functions defined on these ODE trajectories. Imagine a function, let's call it $f$ , that takes a noisy state $x(t)$ at any time $t$ along a particular PF ODE trajectory and directly maps it back to the origin of that trajectory, $x(0)$ .

Mathematically, let $\{x(t)\}_{t \in [t_{\text{min}}, T]}$ be the solution trajectory of the PF ODE starting from $x(0)$ , where $t_{\text{min}}$ is a small positive value close to zero (acting as the lower bound of integration) and $T$ is the maximum noise schedule time. A function $f(x, t)$ satisfies the consistency property if, for all $t$ within the interval $[t_{\text{min}}, T]$ along this specific trajectory:

f(x(t), t) = x(0)

This equation shows that no matter where you are on the trajectory (at any time $t > t_{\text{min}}$ ), the consistency function $f$ always yields the initial data point $x(0)$ from which that trajectory originated.

Furthermore, this implies a form of self-consistency: for any two points $x(t_1)$ and $x(t_2)$ on the same trajectory (where $t_1, t_2 \in [t_{\text{min}}, T]$ ):

f(x(t_1), t_1) = f(x(t_2), t_2) = x(0)

It's this characteristic self-consistency across time for points on the same trajectory that lends the name to these models.

Diagram illustrating the consistency property. Points $x(t_{min})$ , $x(t_1)$ , $x(t_2)$ , and $x(T)$ lie on the same Probability Flow ODE trajectory originating from $x(0)$ . The consistency function $f(x, t)$ maps each of these points back to the original data point $x(0)$ .

Implications for Sampling

The existence of such a consistency function $f$ has profound implications for generation. If we could learn or accurately approximate $f(x, t)$ , we could perform generation in potentially a single step. We would start with a sample $x_T$ drawn from the noise distribution $p_T(x)$ (typically a standard Gaussian) and simply compute:

\hat{x}_0 = f(x_T, T)

This directly estimates the data point $x_0$ corresponding to the trajectory ending at the noise sample $x_T$ , bypassing the need for iterative refinement used in DDPM, DDIM, or other ODE solvers.

Contrast with Score-Based Sampling

This is fundamentally different from standard diffusion model sampling. Traditional methods learn a model (often a U-Net or Transformer) to predict the score function $\nabla_x \log p_t(x)$ or the noise $\epsilon$ added at time $t$ . Sampling then involves using these predictions within a numerical solver (like Euler-Maruyama for the SDE or DDIM/DPM-Solver for the ODE) to take many small steps $\Delta t$ backward in time, gradually transforming noise $x_T$ into a sample $\hat{x}_0$ .

In contrast, consistency models aim to learn the result of the entire integration process from time $T$ back to $t_{\text{min}} \approx 0$ , encapsulated within the function $f$ . The subsequent sections will explore how we can train neural networks to approximate this consistency function $f(x, t)$ , enabling extremely fast generation.

Was this section helpful?

References

Consistency Models, Yang Song, Prafulla Dhariwal, Mark Chen, Ilya Sutskever, 2023 International Conference on Machine Learning (ICML) DOI: 10.48550/arXiv.2303.01469 - Introduces the consistency model framework, the core idea of one-step generation, and the consistency property itself. This is the foundational paper for the section's topic.
Score-Based Generative Modeling through Stochastic Differential Equations, Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole, 2021 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.2011.13456 - Establishes the continuous-time formulation of diffusion models using Stochastic Differential Equations (SDEs) and Probability Flow ODEs, providing the theoretical basis for the ODE trajectories discussed.
Denoising Diffusion Probabilistic Models, Jonathan Ho, Ajay Jain, Pieter Abbeel, 2020 Advances in Neural Information Processing Systems (NeurIPS), Vol. 33 DOI: 10.48550/arXiv.2006.11239 - A seminal paper popularizing modern diffusion models, illustrating the iterative sampling process that consistency models aim to circumvent. Provides context for the 'standard diffusion samplers' mentioned.
Elucidating the Design Space of Diffusion-Based Generative Models, Tero Karras, Miika Aittala, Timo Aila, Samuli Laine, 2022 Advances in Neural Information Processing Systems (NeurIPS), Vol. 35 DOI: 10.48550/arXiv.2206.00364 - Provides a unified theoretical and practical framework for various diffusion models, including insights into the ODE perspective and sampling strategies that lead to fast inference.