Consistency Models

Table of Contents

1. Introduction

1. Introduction

Consistency models are a new family of models based on continuous-time diffusion models [Song et al. ICLR 2021], [Karras el al. NeurIPS 2022] that achieve 1-step generation.

2. Background

Continuous-time diffusion models [Song et al. ICLR 2021] are formulated with the stochastic differential equation (SDE): $$\begin{array}{}\text{(Eq. 1)}& \text{d}{\mathbf{x}}_t=\underset{\text{Deterministic Term}}{\underbrace{\mathit{\mu }({\mathbf{x}}_t,t)\text{d}t}}+\underset{\text{Stochastic Term}}{\underbrace{\sigma (t)\text{d}{\mathbf{w}}_t}},\end{array}$$ where:

• $\sigma (\cdot )$ is the noise schedule.
• $\mathit{\mu }(\cdot ,\cdot )$ is the drift function.
• $\{{\mathbf{w}}_t{\}}_{t\in [0,T]}$ is a Wiener process or Brownian motion.

Being $p({\mathbf{x}}_t)$ the distribution of ${\mathbf{x}}_t$, note that $p_0(\mathbf{x})\equiv p_{data}(\mathbf{x})$.
From the SDE, and with the help of the Fokker-Planck equation, we can derive the Probability Flow ODE (PF-ODE) (for the derivation of [Eq. 2] check [Song et al. ICLR 2021]): $$\begin{array}{}\text{(Eq. 2)}& \text{d}{\mathbf{x}}_t=[\mathit{\mu }({\mathbf{x}}_t,t)\text{d}t-\frac{1}{2}\sigma (t{)}^2\mathrm{\nabla }\log p_t(\mathbf{x})]\text{d}t,\end{array}$$ [Karras el al. NeurIPS 2022]

3. Consistency Models

$$\begin{array}{}\text{(Eq. X)}& {\mathcal{L}}_{CD}^N(\mathit{\theta },{\mathit{\theta }}^{-};\varphi )=\mathbb{E}[\lambda (t_n)d({\mathit{f}}_{\mathit{\theta }}({\mathbf{x}}_{t_{n\ast 1}},t_{n+1}),{\mathit{f}}_{\mathit{\theta }\mathbf{-}}({{\hat{\mathbf{x}}}^{\varphi }}_{t_n},t_n))]\end{array}$$ rember that the expectation $\mathbb{E}$ is like a mean but without samples, so it is a theoretical mean of the distribution.