1Convolutional Neural Networks 2Recurrent Neural Networks 3LSTMs 4Autoencoders & VAEs 5GANs 6Skip Connections & ResNets

Lensa ML

How Skip Connections Work

Step 1 of 6: Deep Network Problem

Why adding layers hurts performance

Network Depth: 20 layers

Depth

20L

Train err

4.8%

Test err

6.7%

Degradation

0.0%

At 20 layers or fewer, the network trains well. Training and test error both converge to low values. This is the baseline.

Drag the Network Depth slider from 10 to 56 layers. The curves follow Figure 1 of He et al. (2015), where a plain 56-layer network had higher training and test error on CIFAR-10 than a 20-layer one. The paper's argument is a construction: copy the 20-layer net and make the extra 36 layers identity mappings, and the deeper net matches the shallow one. So a solution at least as good exists, but gradient descent fails to find it. The plain nets already used batch normalization, so vanishing gradients alone do not explain the gap.

Step 2 of 6: The Skip Connection

F(x) + x: the key insight

Input x: 0.50

Input x

0.50

F(x)

0.700

F(x) + x

1.200

The skip connection adds x directly to F(x). Even if F(x) learned nothing useful, the output would still be x: the identity mapping.

Drag the Input x slider from 0 to 2 and compare F(x) with F(x) + x. In a real ResNet, x is a whole feature map and F is two or three convolution layers with batch normalization and ReLU. The addition is element-wise, so F(x) must have the same shape as x. When a block halves the spatial size or doubles the channels, the shortcut uses a 1x1 convolution with stride 2 to match shapes, called a projection shortcut. An identity skip adds no parameters at all.

Step 3 of 6: Residual Learning

Learning the difference is easier

Target output: 1.00

Target

1.00

Identity (x)

0.80

Residual

0.20

The network only needs to learn the small difference (residual) between the target and the identity. Learning a small correction is much easier than learning the full mapping.

Drag the Target output slider from 0 to 2. The input is fixed at x = 0.8, so a target of 1.0 needs only F(x) = 0.2. Weights start small, so F(x) starts near zero and a fresh residual block behaves almost like the identity. Each block then only has to learn a correction. Some training recipes push this further: setting the last batch-norm scale in each block to zero makes every block an exact identity at the start, which speeds up training of very deep ResNets. The residual stream in Transformers follows the same reasoning.

Step 4 of 6: Gradient Highway

Why gradients survive in ResNets

Network Depth: 8 layers

Depth

Grad (no skip)

0.0037

Grad (skip)

0.6983

Notice how gradients vanish without skips (left) but stay healthy with skips (right). A plain layer multiplies the gradient by its local derivative (here 0.45). A residual block computes x + F(x), so it multiplies the gradient by 1 + F′; the residual branch starts small (here F′ = −0.05), so the factor stays near 1. The + gives gradients a direct 'highway' backward.

Drag Network Depth from 2 to 20 and compare the two bar charts. The math is the chain rule. For y = x + F(x), ∂y/∂x = 1 + ∂F/∂x, so the gradient reaching an early layer is a product of terms (1 + ∂F/∂x) instead of a product of ∂F/∂x alone. Expanding that product gives a sum that always includes a direct term of 1. Veit et al. (2016) showed that a ResNet behaves like an ensemble of many shallower paths, and most of the gradient flows through paths only 10 to 34 layers long.

Step 5 of 6: ResNet Block

Basic vs. bottleneck architecture

Block Type: Basic

Params (basic)

1180K

Params (bottleneck)

70K

Saving at 256 ch

94%

The basic block uses two 3x3 convolutions with a skip connection. Used in ResNet-18 and ResNet-34, it's simple and effective for shallower architectures.

Drag the Block Type slider from Basic to Bottleneck. At 256 channels, two 3x3 convolutions need 2 × 256 × 256 × 9 ≈ 1.18 million weights. The bottleneck version (1x1 down to 64, 3x3 at 64, 1x1 back to 256) needs 16,384 + 36,864 + 16,384 = 69,632. A follow-up paper by the same authors (He et al., 2016) moved batch normalization and ReLU before each convolution. In this pre-activation design the shortcut is a pure identity from input to output, and it let them train a 1,001-layer ResNet.

Step 6 of 6: Going Deeper

From 18 to 101 layers

Architecture: ResNet-18

Blocks

Params

11.7M

Top-1 acc

69.8%

ResNet-18 uses basic blocks (two 3x3 convs each). With 11.7M parameters, it is fast and a common baseline. Skip connections make even this 'shallow' ResNet train reliably.

Drag the Architecture slider from ResNet-18 to ResNet-101 and watch BLOCKS, PARAMS, and TOP-1 ACC. ResNet-152, the deepest ImageNet model in the paper, was part of the ensemble that won the 2015 ImageNet classification challenge with 3.57% top-5 error. Skip connections then spread well beyond classification. DenseNet concatenates earlier feature maps instead of adding them, U-Net links matching encoder and decoder layers for segmentation, and every Transformer block wraps attention and its feed-forward layer in a residual connection.

What are skip connections (residual connections)?

A skip connection is a shortcut that adds a layer block's input directly to its output, so the block computes F(x) + x instead of F(x). The layers then only have to learn the residual F(x), the difference between the desired output and the input. This simple addition is the core of residual networks (ResNets), introduced by He et al. in 2015.

Skip connections are what make very deep networks trainable. They let ResNets go past 100 layers in image classification, and they appear in U-Net for segmentation, in detection backbones, and in every Transformer block, where attention and the feed-forward layer each sit inside a residual connection. Without them, stacking more layers often makes training error worse instead of better.

Key ideas

A skip connection computes F(x) + x, so the layers learn only the residual F(x).
Plain networks get worse training error as they grow deeper, even with batch normalization.
Because ∂(x + F(x))/∂x = 1 + ∂F/∂x, gradients always have a direct path back.
A block whose F(x) is near zero acts as the identity, so extra blocks are cheap to add.
Bottleneck blocks use 1x1 convolutions to cut parameters in deep ResNets.
Residual connections are used in ResNets, U-Nets, and every Transformer block.

Common questions

What is the difference between a skip connection and a residual connection?

A skip connection is any path that carries a layer's input past one or more layers. A residual connection is the specific kind that adds the input to the output, as in ResNet. Skip connections can also concatenate, as in DenseNet and U-Net, or pass through a learned gate, as in highway networks.

Do skip connections reduce overfitting?

Not directly. They mainly fix an optimization problem: deeper plain networks fail to fit even the training data. Overfitting is still handled by data augmentation, weight decay, and other regularization. Deeper ResNets do generalize well in practice, but that comes from being trainable at all.

What is a highway network?

A highway network (Srivastava et al., 2015) is an earlier design where a learned gate T(x) mixes the layer output and the input: y = T(x) · H(x) + (1 − T(x)) · x. It made networks with about 100 layers trainable. ResNet removed the gate and always adds the input, which is simpler and worked better at large scale.

What does the number in ResNet-50 mean?

It counts the layers with learned weights along the main path: the first 7x7 convolution, the convolutions in all residual blocks, and the final fully connected layer. ResNet-50 has 16 bottleneck blocks of 3 convolutions each, so 1 + 48 + 1 = 50. Shortcuts, pooling, and batch normalization are not counted.

Related explainers

Convolutional Neural Networks — How CNNs see and understand images
Vanishing & Exploding Gradients — Why deep networks struggle and how to fix it
Batch Normalization — Why normalizing activations speeds up deep network training
Backpropagation — How networks learn from their mistakes
The Transformer Architecture — A visual guide to 'Attention Is All You Need'
Image Segmentation — Classify every pixel in an image — from semantic regions to individual object instances

References

Last updated October 25, 2026