How Skip Connections Work
Deep Network Problem
Why adding layers hurts performance
DEPTH
20L
TRAIN ERR
5.6%
TEST ERR
7.5%
DEGRADATION
0.0%
At 20 layers or fewer, the network trains well. Both training and test error converge to low values — this is the baseline.
Step 1 of 6: Deep Network Problem
Why adding layers hurts performance
DEPTH
20L
TRAIN ERR
5.6%
TEST ERR
7.5%
DEGRADATION
0.0%
At 20 layers or fewer, the network trains well. Both training and test error converge to low values — this is the baseline.
Step 2 of 6: The Skip Connection
F(x) + x: the key insight
INPUT x
0.50
F(x)
0.700
F(x) + x
1.200
The skip connection adds x directly to F(x). Even if F(x) learned nothing useful, the output would still be x -- the identity mapping.
Step 3 of 6: Residual Learning
Learning the difference is easier
TARGET
1.00
IDENTITY (x)
0.80
RESIDUAL
0.20
The network only needs to learn the small difference (residual) between the target and the identity. Learning a small correction is much easier than learning the full mapping.
Step 4 of 6: Gradient Highway
Why gradients survive in ResNets
DEPTH
8
GRAD (NO SKIP)
0.0037
GRAD (SKIP)
0.7021
Notice how gradients vanish without skips (left) but stay healthy with skips (right). The skip connection provides a 'gradient highway' -- a direct path for gradients to flow backward through the + operation.
Step 5 of 6: ResNet Block
Basic vs. bottleneck architecture
PARAMS (BASIC)
1180K
PARAMS (BOTTLENECK)
70K
COMPRESSION
94%
The basic block uses two 3x3 convolutions with a skip connection. Used in ResNet-18 and ResNet-34, it's simple and effective for shallower architectures.
Step 6 of 6: Going Deeper
From 18 to 101 layers
BLOCKS
8
PARAMS
11.7M
TOP-1 ACC
69.8%
ResNet-18 uses basic blocks (two 3x3 convs each). With just 11.7M parameters, it's fast and a great baseline. Skip connections make even this 'shallow' ResNet train reliably.