ENID
Lensa ML

Diffusion Models

Step 1 of 6: Forward Diffusion

Blend the data with Gaussian noise until nothing is left

an image, same noise
Step t0
Signal √ᾱ
1.00
Noise √(1−ᾱ)
0.00
Signal² + noise²
1.00

These 260 points are the training data, two moon shapes. The goal of a diffusion model is to generate new points that look like they came from the same shapes. Training starts by destroying the data on purpose: drag t or press Add noise.

Drag t from 0 to 1000 and watch the moons dissolve. The forward process has no learned parameters: x_t = √ᾱ_t·x₀ + √(1 − ᾱ_t)·ε, with ε drawn from a standard Gaussian. Because (√ᾱ)² + (√(1 − ᾱ))² = 1, the overall spread stays about the same while the structure disappears. Any step can be sampled straight from x₀ without running the steps before it, which is what makes training cheap. Image models do the same to every pixel, so an image is just a point with millions of coordinates.

Step 2 of 6: Noise Schedule

How fast the signal disappears

00.51ᾱ_t (share of signal energy) over 1000 steps
Step t500
ᾱ linear
0.079
ᾱ cosine
0.494
Steps that are ~pure noise (linear)
327 / 1000

The schedule decides how fast ᾱ falls. The original linear schedule destroys signal quickly: by t = 500 only 0.08 of it is left, and 327 of the 1000 steps (ᾱ below 0.01) are spent on images that are already pure noise. The cosine schedule keeps 0.49 at t = 500 and wastes only 65 steps, so training sees many more useful noise levels. At t = 500, linear leaves ᾱ = 0.079.

Switch between the two schedules and compare the curves. DDPM (Ho et al., 2020) used a β that rises linearly from 0.0001 to 0.02 over 1000 steps, so ᾱ falls fast and many late steps are nearly pure noise. The cosine schedule (Nichol and Dhariwal, 2021) shapes ᾱ like a cosine so the signal fades more evenly. Latent diffusion models such as Stable Diffusion use yet another variant, and some newer models train on a continuous noise level instead of integer steps.

Step 3 of 6: Training Objective

Learn to predict the noise that was added

x₀x_tx̂₀ε vs ε̂— true noise ε- - predicted ε̂(x_t, t)best possible loss at each step t (0 → 1000)
Step t250
This example's loss |ε − ε̂|²
0.84
Average at this t
1.23
x̂₀ off by
0.39

Training picks a data point x₀, a step t and noise ε, builds x_t, and asks the network for ε̂(x_t, t), the noise it thinks was added. The loss is |ε − ε̂|². Here ε̂ comes from the exact formula for this toy data, the best any network could do. At low noise the loss is high (1.23 on average): x_t is close to x₀, but many nearby points on the moon could have produced it, so the exact noise is a guess. Predicting the noise is equivalent to predicting the clean point: x̂₀ = (x_t − √(1−ᾱ)·ε̂)/√ᾱ.

Change t and draw new examples. The DDPM loss is the average of |ε − ε̂(x_t, t)|² over random data points, steps, and noise, the simple objective that worked better than the full variational bound. Even a perfect network cannot reach zero loss at low noise, because several clean points could explain the same x_t; the curve shows that floor. Predicting ε, x₀, or the velocity v = √ᾱ·ε − √(1 − ᾱ)·x₀ are equivalent parameterizations, and different models choose different ones.

Step 4 of 6: Reverse Process

The denoiser is a field of arrows toward the data

Click anywhere to drop a point and watch where denoising carries it.

Step t300
Noise level √(1−ᾱ)
0.46
Average pull
0.92
Your point lands on
bottom moon

Each arrow is the trained network's answer at that spot: "remove this much noise", which points toward where the data probably came from (the faint dots show the data, shrunk by √ᾱ). At lower noise the arrows bend toward the two moons: detail appears only late in denoising. Following the arrows step by step while t shrinks is the reverse process. Your point slides along the orange path and ends on the bottom moon, a place no training point had to be.

Move t and click to drop points. Each arrow is −√(1 − ᾱ)·ε̂, a scaled score: the direction in which the noisy data distribution gets denser, which is why diffusion models are also called score-based models (Song and Ermon, 2019). At high noise the field only points toward the middle of the data, and the moons appear in the arrows only at low noise, the same way image models settle the layout first and fine texture last. Here the arrows come from the exact formula for the toy distribution; a real model learns them with a neural network.

Step 5 of 6: Sampling from Noise

Fresh noise in, a new sample out

t = 0grey = training data · blue = generated
Network calls
25
On the moons
100%
Blobs covered
100%
Dist. to nearest training point
0.047

100% of the generated points sit on the moons and they cover 100% of the shape, yet they are new: the typical one is 0.047 away from the nearest training point. That's generation, not reconstruction. Press New noise for a different set. DDPM adds fresh noise each step; DDIM removes it deterministically, so the same starting noise always gives the same result, which helps when steps are few.

Compare step counts and the two samplers, and draw new noise. DDPM sampling adds a little fresh noise at every step and originally needed about 1000 steps. DDIM (Song et al., 2020) runs the same trained model deterministically and works well with tens of steps, and later solvers and distilled models get good samples in 1 to 4 steps. Every step costs one full network evaluation, which is why diffusion is slower than a single-pass generator. Latent diffusion, used by Stable Diffusion, saves more time by running the process on a compressed latent image instead of full-resolution pixels.

Step 6 of 6: Guidance Scale

Trade variety for matching the prompt

ε = ε_uncond + 3.0 × (ε_cond − ε_uncond)
Guidance w3.0
In the requested moon
100%
Its blobs covered
92%
Spread (std)
0.84

At w = 3.0 the step is pushed past the conditional answer, away from "any data" and toward "this prompt". Samples crowd into the most typical part of the moon: coverage drops to 92% and the spread to 0.84. Image models use w around 5 to 10 because it makes images match the prompt better, at the cost of variety.

Pick a moon and raise w. Classifier-free guidance (Ho and Salimans, 2022) trains one network that sometimes sees the prompt and sometimes a blank, then combines the two predictions at sampling time as ε_uncond + w·(ε_cond − ε_uncond). w = 1 is ordinary conditional sampling. A larger w exaggerates whatever distinguishes the prompt, so samples match it more closely but vary less, and a very large w gives oversaturated, unnatural images. Guidance doubles the cost of each step, one more reason fast samplers matter.

What is a diffusion model?

A diffusion model is a generative model that learns to turn random noise into realistic data by reversing a gradual noising process. During training, real images are corrupted with Gaussian noise at random strengths, and a neural network learns to predict that noise. To generate, the model starts from pure noise and removes the predicted noise a little at a time until an image appears.

The approach was introduced by Sohl-Dickstein et al. in 2015 and made practical by Denoising Diffusion Probabilistic Models (DDPM, Ho et al., 2020). Diffusion models power text-to-image systems such as Stable Diffusion, DALL·E 2, and Imagen, and are also used for audio, video, and molecule generation.

Key ideas

  • The forward process adds Gaussian noise on a fixed schedule; nothing in it is learned.
  • The network learns one task: predict the noise in a noisy image at step t.
  • Generation starts from fresh noise and removes predicted noise step by step, so it produces a new sample, not a stored one.
  • The noise schedule controls how quickly the signal disappears.
  • Fewer sampling steps are faster; methods like DDIM keep quality with far fewer steps.
  • Classifier-free guidance trades diversity for closer prompt matching.

Short takes on this topic

Common questions

How are diffusion models different from GANs?

A GAN generates an image in one pass and is trained by pitting a generator against a discriminator, which can be unstable and can miss parts of the data. A diffusion model is trained with a simple regression loss and generates over many steps. It is slower to sample but more stable to train and usually covers the variety in the data better.

How does a text prompt control the image?

The prompt is turned into vectors by a text encoder, such as CLIP's or T5's. The denoising network reads those vectors through cross-attention at every step, so the noise it predicts depends on the prompt.

Why predict the noise instead of the clean image?

Given the noisy image and the step t, either one can be computed from the other, so the choices are mathematically equivalent. In practice, predicting the noise gave better results in DDPM. Some later models predict a mix of the two, called v-prediction.

Are diffusion models only used for images?

No. The same method generates audio, video, 3D shapes, and molecular and protein structures. Anything that can be represented as numbers and gradually noised can be modeled this way.

Related explainers

References

  1. Sohl-Dickstein et al. (2015), "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"
  2. Ho, Jain & Abbeel (2020), "Denoising Diffusion Probabilistic Models"
  3. Song, Meng & Ermon (2020), "Denoising Diffusion Implicit Models"
  4. Nichol & Dhariwal (2021), "Improved Denoising Diffusion Probabilistic Models"
  5. Rombach et al. (2022), "High-Resolution Image Synthesis with Latent Diffusion Models"
  6. Ho & Salimans (2022), "Classifier-Free Diffusion Guidance"

Last updated