1Convolutional Neural Networks 2Recurrent Neural Networks 3LSTMs 4Autoencoders & VAEs 5GANs 6Skip Connections & ResNets

Lensa ML

How Autoencoders & VAEs Work

Step 1 of 6: The Intuition

What does an autoencoder learn?

Smile0.4

Eye size0.0

Face width0.0

Glasses-0.6

Attributes

Active

Dims

An autoencoder compresses high-dimensional data (like a face) into a small set of latent attributes. The decoder reconstructs the original from just these few numbers. Drag the sliders to see how each attribute controls a different aspect of the face.

Drag the Smile, Eye size, Face width, and Glasses sliders. Here 4 numbers stand in for a whole face. A real autoencoder learns its latent dimensions from data, and they rarely line up this neatly with named features. The only training signal is reconstruction error, such as the mean squared difference between the input x and the output x̂. The narrow bottleneck forces compression: if the code were as large as the input, the network could pass it through unchanged. A linear autoencoder trained with squared error learns the same subspace as PCA.

Step 2 of 6: VAE Architecture

Encoding as distributions, not points

The VAE encoder outputs two vectors: μ (mean) and σ (standard deviation), which define a distribution for each latent dimension instead of a single point. The decoder receives a sample from it, drawn with the reparameterization trick: take ε from N(0,1) and compute z = μ + σ · ε.

Follow the network from left to right: 8 inputs, hidden layers of 6 and 4, then μ and σ with 3 values each, a sampled z of 3, and a mirrored decoder of 4, 6, and 8. In practice the encoder outputs log σ² rather than σ, because a log variance can be any real number while σ must stay positive; σ = exp(½ log σ²). After training, generation needs only the decoder: sample z from N(0, 1) and decode it. Kingma and Welling introduced this model in 2013.

Step 3 of 6: Statistical Motivation

Why variational inference?

Given a photo of a smiling person (x), the goal is to infer the latent features (z), such as smile or hair color, that produced it. Bayes' rule gives:

p(z|x) = p(x|z) · p(z) / p(x)

Computing p(x) means integrating over every possible combination of latent features, which is intractable in practice. Instead, a neural network (the encoder) learns a simpler approximation q(z|x). Try fitting it below.

μ0.20

σ0.40

KL(q‖post)

0.116

KL(q‖prior)

0.516

Fit

good

The VAE loss function balances two goals:

1. Reconstruct well

The decoder should be able to rebuild the original input from the latent code z

2. Stay organized

The encoder's output q(z|x) should stay close to a simple bell curve N(0,1)

Loss = reconstruction error + KL(q(z|x) ‖ p(z))

Good fit. Your q(z|x) closely matches the true posterior. In a real VAE the encoder finds μ and σ automatically, which is what you just did by hand.

Drag the μ and σ sliders and watch the KL(q‖post) and KL(q‖prior) cards. A VAE cannot measure KL to the true posterior directly, because that needs p(x). It maximizes the evidence lower bound (ELBO) instead: log p(x) ≥ E_q[log p(x|z)] − KL(q(z|x) ‖ p(z)). The gap in that inequality is exactly KL to the true posterior, so raising the bound pulls q toward it. For a Gaussian q against N(0, 1) the KL term has a closed form, ½(μ² + σ² − 1 − ln σ²): about 0.52 at μ = 0.2, σ = 0.4.

Step 4 of 6: Reparameterization Trick

Making sampling differentiable

ε0.00

0.35

0.60

0.00

z = μ+σε

0.35

When ε ≈ 0, z stays close to μ, so the sample lands near the mean. The randomness from ε is external to the network, so gradients still flow through μ and σ during backpropagation.

Drag the ε slider from −3 to 3. With μ = 0.35 and σ = 0.6, ε = 1 gives z = 0.35 + 0.6 × 1 = 0.95. A real ε is drawn from N(0, 1), and the slider's ±3 range covers about 99.7% of those draws. Before this trick, gradients through random samples needed score-function estimators such as REINFORCE, which are much noisier. The same idea appears wherever a model has to sample and still train by backpropagation, for example the Gumbel-softmax trick for discrete choices.

Step 5 of 6: Latent Space

Visualizing the effect of β

Picture a VAE trained on handwritten digits (0–9). The encoder compresses each 28×28 pixel image into just 2 latent dimensions (z₁, z₂). Each color below represents a different digit class. Drag β to see how it shapes this 2D space. The points are a schematic drawn to show the typical effect, not codes from a trained model.

↑ reconstruction loss↑ KL divergence (weighted by β)

β1.00

1.00

Mode

balanced

Clusters

With β ≈ 1.0, both losses balance. Clusters are distinct enough to separate digits, but they spread out and overlap smoothly. With no dead zones, the space moves gradually from one digit to the next, so new samples look realistic.

Drag β from 0 to 5 and watch the Mode card switch between recon, balanced, and kl. A standard VAE uses β = 1. Higgins et al. (2017) proposed the β-VAE, with β above 1, to push each latent dimension toward a separate factor of variation such as stroke width or slant. The cost is blurrier reconstructions. The collapsed state at high β is called posterior collapse: the decoder learns to ignore z. It is most common with powerful decoders, and KL annealing, which ramps β up from 0 during training, is a standard fix.

Step 6 of 6: Generation

Smooth interpolation in latent space

0.0

0.2

0.3

0.5

0.7

0.8

1.0

Smile

0.1

Eye size

0.1

Face width

0.0

Glasses

-0.1

0.50

Smile

0.10

Eyes

0.10

Width

0.00

At t = 0.50, each latent attribute is a smooth blend of A and B. The smile fades, the eyes grow, and glasses appear gradually. A continuous latent space makes it possible to generate new, realistic data by sampling or interpolating anywhere.

Drag the slider from A to B. The readouts follow a straight-line blend, z = (1 − t) · z_A + t · z_B, so at t = 0.5 the smile value sits halfway between 0.8 and −0.6, at 0.1. In high-dimensional Gaussian latents, spherical interpolation (slerp) is often preferred, because points on the straight line between two samples have a smaller norm than typical samples. VAE samples tend to be blurry, which is one reason sharper generators such as GANs took over image synthesis.

What is an autoencoder?

An autoencoder is a neural network trained to copy its input to its output through a narrow middle layer, called the bottleneck or latent code. The encoder half compresses the input into a few numbers, and the decoder half rebuilds the input from them. A variational autoencoder (VAE) changes the encoder so it outputs a probability distribution over codes instead of one point, which turns the decoder into a generator of new data.

Autoencoders are used for dimensionality reduction, denoising, anomaly detection (inputs the model reconstructs badly are likely unusual), and learning features without labels. VAEs are used to generate images and molecules and to explore smooth latent spaces. The autoencoder in Stable Diffusion compresses images into a small latent grid so the diffusion model can work on far fewer numbers than raw pixels.

Key ideas

An autoencoder compresses input into a small latent code and reconstructs it from that code.
The bottleneck forces the network to keep only the most useful information.
A VAE encoder outputs a mean μ and standard deviation σ for each latent dimension.
The VAE loss is reconstruction error plus β times the KL divergence from N(0, 1).
The reparameterization trick z = μ + σ · ε makes sampling differentiable.
A smooth latent space lets the decoder generate new data by sampling or interpolating.

Common questions

What is the difference between an autoencoder and a VAE?

A plain autoencoder maps each input to one point and is trained only to reconstruct, so its latent space can have gaps that decode to nonsense. A VAE maps each input to a Gaussian and adds a KL penalty that keeps those Gaussians close to N(0, 1). That fills in the space, so random samples from N(0, 1) decode to realistic outputs.

What is a denoising autoencoder?

A denoising autoencoder receives a corrupted input, for example with random noise or masked pixels, and is trained to output the clean original. This stops it from learning a trivial copy and forces it to learn the structure of the data. Masked autoencoders for images and masked language modeling in BERT follow the same idea.

Why are VAE outputs blurry?

With a Gaussian decoder, the reconstruction term is a squared error, which rewards the average of all plausible outputs. Averaging several sharp images gives a blurry one. Noise in z adds to this, since nearby codes must decode to similar images.

Is an autoencoder supervised or unsupervised?

It needs no labels, so it is usually called unsupervised or self-supervised learning: the input is its own target. The learned encoder is often reused as a feature extractor for a supervised task trained on a smaller labeled set.

Related explainers

KL Divergence — Measuring how different two distributions are
Dimensionality Reduction — PCA and t-SNE — seeing high-dimensional data
GANs — Two networks competing to create
Diffusion Models — Generate images by learning to reverse a gradual noising process
Embeddings — Turning words and categories into vectors
Probability & Distributions — Spam, densities, and the math of uncertainty

References

Last updated October 25, 2026