Consistency Models

Table of Contents

1. Introduction

1. Introduction

Consistency models are a new family of models based on continuous-time diffusion models [Song et al. ICLR 2021], [Karras el al. NeurIPS 2022] that achieve 1-step generation.

2. Background

Continuous-time diffusion models [Song et al. ICLR 2021] are formulated with the stochastic differential equation (SDE): \[ \text{d}\mathbf{x}_t = \underbrace{\boldsymbol{\mu}(\mathbf{x}_t, t) \ \text{d}t}_{\text{Deterministic Term}} + \underbrace{\sigma(t) \ \text{d}\mathbf{w}_t}_{\text{Stochastic Term}}, \label{eq:ode} \tag{Eq. 1} \] where:

• \( \sigma(\cdot) \) is the noise schedule.
• \( \boldsymbol{\mu}(\cdot, \cdot) \) is the drift function.
• \( \{\mathbf{w}_t\}_{t \in [0,T]} \) is a Wiener process or Brownian motion.

Being \( p(\mathbf{x}_t) \) the distribution of \( \mathbf{x}_t \), note that \( p_0(\mathbf{x}) \equiv p_{data}(\mathbf{x}) \).
From the SDE, and with the help of the Fokker-Planck equation, we can derive the Probability Flow ODE (PF-ODE) (for the derivation of [Eq. 2] check [Song et al. ICLR 2021]): \[ \text{d}\mathbf{x}_t = \left[ \boldsymbol{\mu}(\mathbf{x}_t, t) \ \text{d}t - \frac{1}{2}\sigma(t)^2 \ \nabla \log p_t(\mathbf{x}) \right] \text{d}t , \label{eq:pf-ode} \tag{Eq. 2} \] [Karras el al. NeurIPS 2022]

3. Consistency Models

\[ \mathcal{L}_{CD}^N(\boldsymbol{\theta}, \boldsymbol{\theta}^{-}; \phi) = \mathbb{E} [\lambda(t_n) d(\boldsymbol{f_\theta}(\mathbf{x}_{t_{n*1}}, t_{n+1}), \boldsymbol{f_{\theta{-}}}(\mathbf{\hat{x}^\phi}_{t_n}, t_n))] \label{eq:loss} \tag{Eq. X} \] rember that the expectation \( \mathbb{E} \) is like a mean but without samples, so it is a theoretical mean of the distribution.