Diffusion Models Explained: How AI Turns Noise Into Images

In short: a diffusion model is an AI model that creates an image (or video, or audio) by starting from pure random noise and removing that noise a little at a time until a picture appears. It learns to do this by being trained on the reverse: real images with more and more noise added. The idea was published in 2015, became practical with UC Berkeley's 2020 DDPM paper, and runs today's best-known image and video generators, including Stable Diffusion and OpenAI's Sora. Many newer models use a close relative called flow matching or rectified flow.

What is a diffusion model, in simple words?

A diffusion model makes a picture by cleaning up noise, step by step. Imagine an old TV showing static. A diffusion model looks at the static and makes it very slightly less random, in a direction that looks a bit more like a real photo. It repeats this many times. After enough steps, the static has become an image.

Each step is small and easy; the skill is in knowing which direction "less noisy" points. That is what training teaches.

How does a diffusion model work, step by step?

Training destroys images with noise; generation runs that process backwards. The 2015 paper that introduced the approach, Deep Unsupervised Learning using Nonequilibrium Thermodynamics (Sohl-Dickstein and colleagues, Stanford and UC Berkeley), describes it as: "systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process", then "learn a reverse diffusion process that restores structure in data".

1. Forward process (training only). Take a real image and add a small amount of random noise. Repeat, many times, until nothing but noise is left.
2. Learning. Show the model a noisy image and ask it to predict the noise that was added. A model that can do this well knows how to make any noisy image a little cleaner.
3. Reverse process (generation). Start from fresh random noise. Ask the model to remove a bit of noise. Repeat for every step. The result is a new image that was never in the training data.

The name comes from physics: the forward process is like a drop of ink diffusing through water until it is evenly spread, and the model learns to run that film in reverse.

How does a text prompt control the image?

The prompt is turned into numbers and fed to the model at every denoising step. The 2021 Latent Diffusion paper behind Stable Diffusion added cross-attention layers so the model can be steered by "general conditioning inputs such as text or bounding boxes". At each step the model asks, in effect, "which direction makes this look more like a red bicycle on a beach?"

A second technique, classifier-free guidance (Ho and Salimans, Google, 2022), makes the prompt matter more. The model is trained both with and without the prompt, and at generation time the two predictions are combined to trade off "sample quality and diversity". The "guidance scale" or "CFG" slider in many image tools sets that trade-off: higher values follow the prompt more strictly, lower values give more varied results.

For how text becomes numbers, see tokens and context windows and how attention works.

What is latent diffusion, and why does Stable Diffusion use it?

Latent diffusion runs the noise-removal on a compressed version of the image instead of on every pixel. The High-Resolution Image Synthesis with Latent Diffusion Models paper (Rombach and colleagues, LMU Munich, Heidelberg University and Runway, 2021) notes that diffusion directly in pixel space "often consumes hundreds of GPU days and inference is expensive". Their fix:

• An autoencoder first squeezes each image into a much smaller grid of numbers (the latent), and can expand it back.
• The diffusion model works only on that small latent, which is far cheaper.
• At the end, the autoencoder turns the clean latent back into a full-size image.

This is the design Stable Diffusion was built on. See Stable Diffusion 3.5.

How did diffusion models develop?

From a 2015 paper to the models behind Stable Diffusion 3 and Sora by 2024.

Year	Paper	What it added
2015	Sohl-Dickstein et al.	The forward-and-reverse diffusion idea
2020	DDPM (Ho, Jain, Abbeel, UC Berkeley)	High-quality images; a state-of-the-art FID score of 3.17 on CIFAR-10
2021	Latent Diffusion (Rombach et al.)	Diffusion in a compressed latent space; text conditioning via cross-attention
2022	Classifier-free guidance (Ho, Salimans)	Stronger prompt control without a separate classifier
2022	Diffusion Transformers (Peebles, Xie)	A transformer in place of the usual U-Net backbone
2022	Flow Matching (Lipman et al., Meta AI and the Weizmann Institute)	A simpler, related way to train noise-to-data models
2024	Rectified flow transformers (Stability AI)	The approach used for Stable Diffusion 3

FID (Fréchet Inception Distance) measures how close generated images are, statistically, to real ones; lower is better.

What are flow matching and rectified flow?

They are close relatives of diffusion that move from noise to image along a straighter path. Classic diffusion takes a curved, many-step route from noise to data. The 2022 Flow Matching paper describes a training method that "subsumes existing diffusion paths as specific instances" and allows more efficient paths with "faster training and sampling". Rectified flow, in the words of Stability AI's 2024 paper, "connects data and noise in a straight line".

In practice the idea is the same - start from noise, refine step by step - so these models are usually grouped with diffusion models. Examples that state it on their own model cards:

• Stable Diffusion 3: Stability AI's paper is titled Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.
• FLUX.1 [dev]: Black Forest Labs' model card calls it "a 12 billion parameter rectified flow transformer". See the current FLUX 3 Image page for the latest version.

How do video models like Sora use diffusion?

They denoise a whole block of video frames at once, as patches in space and time. OpenAI's Sora technical report, Video generation models as world simulators, says it trains "text-conditional diffusion models jointly on videos and images" using "a transformer architecture that operates on spacetime patches of video and image latent codes", and that "diffusion transformers scale effectively as video models as well". See Sora 2.

How is a diffusion model different from an LLM?

An LLM writes one token at a time from left to right; a diffusion model refines the whole output at once over many steps.

	Large language model	Diffusion model
Typical output	Text	Images, video, audio
How it generates	Predicts the next token, then the next	Removes noise from the whole output, step by step
What one step produces	One more token	A slightly cleaner version of everything
More detail	How LLMs work	This guide

The two are often combined: a chatbot can pass your request to an image model, and many image models use a language model to read the prompt. Current image and video model rankings are on the AI model rankings page.

Words to know

• Noise: random values, like TV static, that the model starts from and gradually removes.
• Denoising step: one pass in which the model makes its input slightly less noisy.
• Forward process: adding noise to real images during training.
• Reverse process: removing noise to generate a new image.
• Latent: a compressed numeric version of an image that the model works on instead of raw pixels.
• Guidance scale (CFG): how strongly the output is pushed to follow the prompt.
• U-Net: the image-processing network used as the backbone of early diffusion models.
• Diffusion transformer (DiT): a diffusion model whose backbone is a transformer.
• Flow matching / rectified flow: related methods that move from noise to data along a straighter path.

Related guides

• Computer vision explained
• Vision Transformer (ViT) explained
• How LLMs work
• Attention Is All You Need: the transformer paper