Lecture 15: Diffusion Models

Learning to generate images by reversing noise, from pixel-space diffusion to text-conditioned latent diffusion.

1. From reconstructing images to generating them

A classifier answers a question about an observed image, such as whether it contains a cat. A generative model instead describes a distribution from which new images can be sampled. In supervised problems this may mean modeling the joint distribution of images and labels; for unconditional image generation, the target is simply the image distribution p_{\mathrm{data}}(x).

The lecture begins by revisiting three constructions:

• An autoencoder maps an image through an encoder and decoder, \hat x=D(E(x)). Reconstruction alone does not ensure that arbitrary latent vectors decode to plausible images.
• A variational autoencoder (VAE) specifies a prior and a probabilistic decoder. An approximate posterior makes its likelihood objective tractable through a variational bound. Sampling starts with a draw from the prior.
• A generative adversarial network (GAN) learns a generator through a discriminator’s feedback. Its usual training objective compares generated and real samples without evaluating the image likelihood.

Diffusion offers another route: deliberately corrupt training images through a known process, and learn transitions that run in the opposite direction. A complicated image-generation problem becomes many local denoising problems. Sohl-Dickstein et al. introduced this construction; DDPMs developed a particularly effective Gaussian formulation.

Scope. These notes follow the October 29 lecture slides, particularly slides 4-20. Sections 2-3 formalize the diagrams in the lecture. The closed-form derivations in Sections 4-5 and the optional score interpretation are reading supplements, included to make the training procedure understandable without returning to the slides.

2. The forward process: destroy structure in controlled steps

Let x_0\in\mathbb R^d be a training image flattened into a vector. Each intermediate x_t has the same dimension. The subscript t denotes a noise level, not a training iteration. Choose a number of steps T and a variance schedule 0<\beta_t<1. The standard Gaussian forward transition is

q(x_t\mid x_{t-1})= \mathcal N\!\left(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\right).

Here q denotes the fixed corruption process, and I is the d\times d identity matrix. Equivalently,

x_t=\sqrt{1-\beta_t}\,x_{t-1}+\sqrt{\beta_t}\,\epsilon_t, \qquad \epsilon_t\sim\mathcal N(0,I),

with independent noise at each step. Multiplying the previous signal by \sqrt{1-\beta_t} attenuates it while the new noise increases uncertainty. The full conditional path distribution factors as

q(x_{1:T}\mid x_0)=\prod_{t=1}^{T}q(x_t\mid x_{t-1}).

Sampling any noise level directly

Define \alpha_t=1-\beta_t and \bar\alpha_t=\prod_{s=1}^{t}\alpha_s, with \bar\alpha_0=1. Repeatedly substituting the transition gives

\begin{aligned} q(x_t\mid x_0) &=\mathcal N\!\left(x_t;\sqrt{\bar\alpha_t}\,x_0, (1-\bar\alpha_t)I\right),\\ x_t&=\sqrt{\bar\alpha_t}\,x_0+ \sqrt{1-\bar\alpha_t}\,\epsilon, \qquad \epsilon\sim\mathcal N(0,I). \end{aligned}

To check the variance, suppose the accumulated noise at step t-1 has variance 1-\bar\alpha_{t-1}. The next variance is \alpha_t(1-\bar\alpha_{t-1})+\beta_t =1-\bar\alpha_t. Independent Gaussian increments remain Gaussian, which proves the formula by induction.

This identity is crucial for efficient training: one can create a noisy example at a randomly selected step without simulating all preceding steps. The single \epsilon in this formula is the accumulated noise, rather than the last incremental noise \epsilon_t.

If the schedule makes \bar\alpha_T sufficiently small, x_T is approximately standard Gaussian. This is an approximation at a finite number of steps; choosing an inadequate schedule leaves a mismatch between the terminal corrupted distribution and the Gaussian sampling prior.

Figure 1. Controlled loss of signal. An original synthetic image corrupted with the closed-form forward equation. The panels show selected marginal noise levels, not a reverse-model output or a measured generation result.

3. The reverse process: learn a distribution over cleaner images

Knowing how to add noise does not tell us how to remove it from an unknown image. By Bayes’ rule,

q(x_{t-1}\mid x_t) =\frac{q(x_t\mid x_{t-1})q(x_{t-1})}{q(x_t)}.

The marginal distributions in this expression depend on the unknown data distribution. Moreover, several plausible clean images can correspond to a similar noisy observation. Reversal therefore requires learning a conditional distribution, rather than algebraically undoing an individual noise draw.

The model starts from p(x_T)=\mathcal N(0,I) and defines

\begin{aligned} p_\theta(x_{0:T})&=p(x_T)\prod_{t=1}^{T}p_\theta(x_{t-1}\mid x_t),\\ p_\theta(x_{t-1}\mid x_t) &=\mathcal N\!\left(x_{t-1};\mu_\theta(x_t,t),\sigma_t^2 I\right). \end{aligned}

We use a fixed time-dependent variance for this exposition. Other models may learn the covariance. The Gaussian transition is a modeling choice motivated by small forward steps; the exact reverse conditional need not be Gaussian for an arbitrary data distribution and finite step size.

A shared neural network receives both x_t and an encoding of t, since the required correction depends on how much noise is present. In the DDPM formulation, this is an image-to-image denoising network, commonly a U-Net. Its output has the same shape as its image input.

The model’s image density is obtained by integrating out the noisy states:

p_\theta(x_0)=\int p_\theta(x_{0:T})\,dx_1\cdots dx_T.

Generation uses only the learned reverse chain. Draw a fresh x_T and sample successively down to x_0. No training image is supplied at inference time. In particular, an unconditional sample need not reproduce the image from which some training corruption was made.

4. Why training reduces to predicting noise

The lecture’s summary links a variational bound with weighted denoising loss. The following derivation supplies the intermediate steps, using the DDPM parameterization.

A tractable training target

Although q(x_{t-1}\mid x_t) is unknown, conditioning additionally on the training image gives a tractable Gaussian posterior. For t\geq2, completing the square in q(x_t\mid x_{t-1})q(x_{t-1}\mid x_0) yields

q(x_{t-1}\mid x_t,x_0)= \mathcal N\!\left(x_{t-1};\tilde\mu_t(x_t,x_0),\tilde\beta_t I\right), \begin{aligned} \tilde\mu_t(x_t,x_0) &=\frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0 +\frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t,\\ \tilde\beta_t&=\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t. \end{aligned}

There is no contradiction: x_0 is available for constructing the training target, but is never an input to the reverse model during generation.

The variational bound

For a fixed training image, Jensen’s inequality gives a lower bound on \log p_\theta(x_0). Negating it gives an upper bound on the negative log likelihood (NLL):

\begin{aligned} -\log p_\theta(x_0)&\leq\mathcal L_{\mathrm{VLB}}(x_0),\\ \mathcal L_{\mathrm{VLB}}(x_0) &=\mathbb E_{q(x_{1:T}\mid x_0)} \left[\log\frac{q(x_{1:T}\mid x_0)}{p_\theta(x_{0:T})}\right]. \end{aligned}

Rearranging the chain factors separates three kinds of terms:

\begin{aligned} \mathcal L_{\mathrm{VLB}}(x_0) ={}&D_{\mathrm{KL}}\!\left(q(x_T\mid x_0)\,\|\,p(x_T)\right)\\ &+\sum_{t=2}^{T}\mathbb E_{q(x_t\mid x_0)} D_{\mathrm{KL}}\!\left(q(x_{t-1}\mid x_t,x_0)\,\|\, p_\theta(x_{t-1}\mid x_t)\right)\\ &-\mathbb E_{q(x_1\mid x_0)}\log p_\theta(x_0\mid x_1). \end{aligned}

The first term measures terminal-prior mismatch and is constant with respect to \theta when the schedule is fixed. The middle terms train the reverse transitions. The last term is a reconstruction likelihood at the data boundary. For discrete image intensities, specifying that likelihood requires an appropriate discretized observation model if one wants an actual likelihood evaluation.

For fixed \sigma_t^2, the part of a middle KL term that depends on the mean is

\frac{1}{2\sigma_t^2} \|\tilde\mu_t(x_t,x_0)-\mu_\theta(x_t,t)\|_2^2.

Predict noise instead of the reverse mean

Use the forward equation to express x_0=(x_t-\sqrt{1-\bar\alpha_t}\,\epsilon)/\sqrt{\bar\alpha_t}. Substituting this into the posterior mean suggests the parameterization

\mu_\theta(x_t,t)=\frac{1}{\sqrt{\alpha_t}} \left(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}} \epsilon_\theta(x_t,t)\right).

Thus the network predicts the accumulated noise, and a known formula converts that prediction into a reverse mean. Substitution into the preceding squared error gives, for t\geq2,

\begin{aligned} \mathcal L_{t-1} &=\mathbb E_{x_0,\epsilon} \left[w_t\|\epsilon-\epsilon_\theta(x_t,t)\|_2^2\right]+C_t,\\ w_t&=\frac{\beta_t^2} {2\sigma_t^2\alpha_t(1-\bar\alpha_t)}. \end{aligned}

Here C_t does not depend on the network parameters. In practice, the widely used simplified DDPM objective drops this weighting:

\mathcal L_{\mathrm{simple}}= \mathbb E_{x_0,t,\epsilon} \left[\|\epsilon-\epsilon_\theta( \sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon,t)\|_2^2\right].

Here the image is sampled from the data distribution, the timestep uniformly from \{1,\ldots,T\}, and the noise from \mathcal N(0,I), independently.

This is supervised regression with targets generated automatically from the data. It is a reweighted training objective, not literally the exact NLL or the complete variational bound. The t=1 boundary term must also be distinguished from the interior Gaussian KL terms above.

Optional reading connection: the score

The score is the gradient of a log density with respect to its input. For the forward conditional Gaussian,

\nabla_{x_t}\log q(x_t\mid x_0) =-\frac{x_t-\sqrt{\bar\alpha_t}x_0}{1-\bar\alpha_t} =-\frac{\epsilon}{\sqrt{1-\bar\alpha_t}}.

At the population squared-error optimum, the noise predictor estimates \mathbb E[\epsilon\mid x_t]. Averaging the conditional score over x_0\mid x_t then gives the score of the noisy marginal. This explains the connection between noise prediction and denoising score matching; it does not mean the network can identify the exact noise in every individual corrupted example.

5. Training and sampling are different computations

One training update samples clean images, timesteps, and Gaussian noise, then regresses the predicted noise toward the sampled noise. For batch size B, a mean-over-examples loss is B^{-1}\sum_{i=1}^{B}\|\epsilon_i- \epsilon_\theta(x_{t_i}^{(i)},t_i)\|_2^2. Averaging over image coordinates as well changes its scale by 1/d.

Training update:
    sample a batch of clean images x0
    independently sample one timestep t and Gaussian noise epsilon per image
    xt = sqrt(alpha_bar[t]) * x0 + sqrt(1 - alpha_bar[t]) * epsilon
    prediction = noise_network(xt, t)
    loss = batch_mean(sum_over_image_coordinates((epsilon - prediction)^2))
    backpropagate loss and update network parameters

After training, a basic ancestral sampler uses the reverse mean from Section 4:

Sampling (one simple DDPM convention):
    x = sample_standard_normal(image_shape)
    for t = T, T-1, ..., 1:
        prediction = noise_network(x, t)
        mean = (x - beta[t] / sqrt(1 - alpha_bar[t]) * prediction)
               / sqrt(alpha[t])
        if t > 1:
            x = mean + sqrt(beta_tilde[t]) * sample_standard_normal(image_shape)
        else:
            x = mean
    return x

This convention uses posterior variance \sigma_t^2=\tilde\beta_t at interior steps and returns the predicted mean at the final step. It describes a common continuous-valued sampling procedure, rather than implementing the discretized boundary likelihood needed for exact image coding. Other variance choices and samplers are possible.

Training needs only a randomly chosen noise level per example; the basic sampler requires a sequence of network evaluations. That distinction explains why a simple denoising training objective can still lead to expensive generation.

6. Latent diffusion: move the repeated computation out of pixel space

Large images make every denoising evaluation expensive. The lecture therefore asks whether diffusion should run directly on pixels. A latent diffusion model first learns a useful compressed representation with an autoencoder, then models the distribution of those representations.

Figure 2. Two stages of latent diffusion. The image encoder supplies training latents. New-image generation starts from latent noise; the decoder is applied after the denoising chain. The diagram is an original schematic of the lecture's latent-diffusion construction.

Write z_0=E(x) and \hat x=D(z_0). The same corruption equation now acts on a latent tensor:

z_t=\sqrt{\bar\alpha_t}z_0+\sqrt{1-\bar\alpha_t}\epsilon.

Generation draws z_T, denoises to z_0, and decodes once with D. The encoder is unnecessary for generating a fresh image from noise. In the two-stage construction, the autoencoder is trained first and held fixed while learning the latent denoiser. Latent scaling must be consistent between training and sampling.

For a dimensional illustration, compressing an RGB 512\times512 image to a 64\times64\times4 latent changes the number of scalar coordinates from 786,432 to 16,384, a factor of 48. This is a size comparison, not a measured speedup: network width, attention, and sampling steps also determine cost. Compression can discard fine details, so reconstruction quality places a limit on the final decoded result.

Rombach et al. develop this separation between perceptual compression and generative modeling, including conditioning through cross-attention.

7. Text conditioning and the Stable Diffusion idea

Unconditional diffusion samples from a general learned image distribution. Text-to-image generation instead aims at a conditional distribution, p_\theta(x\mid y), where y is a prompt. A text encoder maps the prompt to conditioning features c=\tau(y). The noise predictor receives these features in addition to the noisy latent and timestep:

\epsilon_\theta(z_t,t,c).

The corresponding training target is still the known noise. What changes is the information available to predict it. In a cross-attention implementation, queries come from image features and keys and values come from text features, letting spatial image features attend to prompt tokens.

At generation time, encode the prompt, draw latent Gaussian noise, run the text-conditioned denoising chain, and decode the final latent. This is the combination emphasized by the lecture’s Stable Diffusion slides: compressed image representations plus iterative denoising plus text conditioning. It describes the latent-diffusion architecture discussed in class, without assuming that all image-generation products or model versions use the same internal design.

8. Comparing VAE, GAN, and diffusion objectives

Property	VAE	GAN	Diffusion
Construction	Prior, decoder likelihood, approximate encoder posterior	Prior and generator, trained against a discriminator	Fixed corruption process, prior, learned reverse transitions
Training signal	Reconstruction likelihood and posterior-prior KL	Adversarial discrepancy between real and generated samples	Variational bound or a reweighted denoising objective
Likelihood treatment	Optimize an evidence lower bound	Standard objective does not evaluate likelihood	Bound the likelihood; simple noise MSE changes the weighting
Basic generation path	Sample latent, decode	Sample latent, run generator	Sample noise, repeatedly denoise; decode if latent-based
Typical concern	Posterior approximation and reconstruction quality	Training instability and mode collapse	Sequential sampling cost and schedule/model approximation

The lecture’s labels such as VAE blurriness, GAN realism, or diffusion mode coverage summarize tendencies rather than universal guarantees. Architecture, data, objective, and inference choices affect quality and diversity. A likelihood-based construction does not by itself prove that every data mode will be represented by a fitted model.

9. Checks for understanding

1. Which direction is learned? The reverse transitions are learned. In the formulation here, the forward variance schedule is fixed.
2. Why can training skip forward steps? Gaussian composition gives the closed-form marginal q(x_t\mid x_0).
3. Why can we compute a posterior target but still need a network? The tractable target also conditions on the training image x_0, which is unavailable during generation.
4. Is noise MSE the exact likelihood objective? No. It is a useful reweighting; the full variational bound also contains terminal and reconstruction terms.
5. What does latent diffusion change? The space in which the chain operates. It still requires learning denoising transitions.
6. What does text conditioning change? The reverse prediction can depend on a prompt, so samples target a conditional image distribution.

The central construction is to learn generation through a sequence of manageable conditional predictions. The forward process supplies noisy inputs and known training targets; the learned reverse process turns fresh noise into new samples.