1Pooling & Downsampling 2Object Detection 3Image Segmentation 4Data Augmentation

Lensa ML

Image Segmentation

Step 1 of 6: Segmentation vs Detection

From image-level labels to pixel-level masks

Granularity0.00

Level

Classify

Granularity

Output

1 label

Classification assigns one label to the whole image. It is quick but has no spatial information: it says what is in the scene, not where.

Drag Granularity from 0 to 1 to go from one label, to boxes, to a label for every cell of the 10×10 scene. A segmentation model outputs an H × W × C tensor, one score per class at every pixel, turned into probabilities with a softmax. Training usually uses per-pixel cross-entropy, often combined with Dice loss when a class covers only a few pixels. Quality is measured by IoU for each class, averaged into mean IoU (mIoU). Labels are expensive: each finely annotated Cityscapes image took more than 1.5 hours.

Step 2 of 6: Semantic Segmentation

Every pixel gets a class label — sky, road, car, person

Number of classes3

Classes

Total pixels

100

Largest class

52 px

With 3 classes, the model must learn distinct boundaries between each region. Per-class counts: Sky=42, Ground=52, Object=6.

Drag Number of classes from 2 to 5. With fewer classes, the scene's five labels (sky, tree, road, car, person) are folded into coarser groups, such as car and person into one object class, so some regions share a color. Semantic segmentation gives every pixel a class but does not separate objects: two cars side by side become one car-colored region. Standard benchmarks include PASCAL VOC, with 20 object classes plus background, Cityscapes, with 19 classes of street scenes, and ADE20K, with 150. Medical imaging and land-cover mapping from satellites are common applications.

Step 3 of 6: Encoder-Decoder Architecture

Compress then expand — the U-shaped backbone of segmentation

Layer depth2

Depth

Bottleneck

64ch

Downsample

2×

Depth 2: one downsampling step halves the spatial resolution and raises the channel count. The decoder upsamples back to the original size.

Drag Layer depth from 1 to 4. Each encoder level here halves the resolution while the channel count grows from 3 to 64, 128 and 256. Downsampling widens the receptive field, so each feature sees more context, which is what it takes to tell a road from a gray wall. The decoder then upsamples back to the input size to give one prediction per pixel. Fully Convolutional Networks (Long et al., 2015) first did this by replacing a classifier's dense layers with convolutions. DeepLab keeps resolution higher with dilated (atrous) convolutions instead.

Step 4 of 6: Skip Connections

U-Net's key insight: bridge encoder details to the decoder

Skip connections1

Skip connections

IoU score

0.97

Boundary acc

95%

Skip connections (U-Net style) concatenate encoder features with decoder features at matching resolutions. Fine spatial detail survives, so the thin arm and the boundaries stay sharp and IoU rises from 0.49 to 0.97.

Drag the Skip connections slider between 0 and 1. The IoU and boundary readouts are measured on a toy 32×32 mask: decoding from the 4×4 bottleneck alone loses the thin arm and scores an IoU of 0.49, while adding the 16×16 skip map raises it to 0.97. In U-Net, each decoder level concatenates the encoder feature map of the same size, so the decoder gets coarse semantics from below and sharp edges from the skip. U-Net (Ronneberger et al., 2015) was built for biomedical images and trained well from very few labeled examples, helped by heavy elastic-deformation augmentation. The same U-shaped design is the denoising network inside many diffusion models, including Stable Diffusion.

Step 5 of 6: Instance Segmentation

Same class, different objects — each gets its own mask

Number of instances2

Instances

Total pixels

Class

Same

Unique masks

2 objects of the same class. Semantic segmentation colors both identically. Instance segmentation gives each its own ID and mask, which counting and tracking need.

Drag Number of instances from 1 to 4. Semantic segmentation answers "which class is this pixel?" and instance segmentation also answers "which object?" Panoptic segmentation combines the two: countable things such as cars and people get instance IDs, and uncountable stuff such as sky and road gets class labels only. Mask R-CNN (He et al., 2017) adds a small mask branch to Faster R-CNN that predicts a 28×28 mask inside each detected box. Segment Anything (2023) returns a mask for any object from a click or box prompt.

Step 6 of 6: Upsampling Methods

Nearest, bilinear, or learned — how to grow feature maps back

Upsampling method0

Method

Nearest

Unique values

Smoothness

Low

Nearest neighbor: each output pixel copies the closest input pixel. It is fast and parameter-free but blocky, with visible 2×2 blocks of identical values.

Drag Upsampling method from 0 to 2 to switch between nearest neighbor, bilinear and transposed convolution as a 3×3 grid grows to 6×6. The transposed convolution here uses stride 2 and an example 3×3 kernel, so each input value stamps a scaled copy of the kernel onto the output. The kernel size is not divisible by the stride, so the copies overlap unevenly: on a constant input, output cells would get total weights of 0.9, 0.8 or 0.6 in a checkerboard. Odena et al. (2016) traced checkerboard artifacts to this and recommended resizing first, then applying a normal convolution.

What is image segmentation?

Image segmentation is the computer vision task of assigning a label to every pixel in an image. Semantic segmentation labels each pixel with a class, such as road or sky. Instance segmentation also separates individual objects of the same class, so two cars get two different masks.

Segmentation is used where exact shapes matter: outlining tumors and organs in medical scans, finding drivable road and lane markings for self-driving cars, mapping land cover from satellite images, and cutting out people for portrait mode and video backgrounds. Most models follow an encoder-decoder design such as U-Net, and promptable models like Segment Anything now produce masks for objects they were never trained on.

Key ideas

Segmentation predicts a class for every pixel, so the output has the same height and width as the image.
Semantic segmentation labels classes, and instance segmentation also separates individual objects.
An encoder downsamples to gather context, and a decoder upsamples back to full resolution.
Skip connections pass fine spatial detail from the encoder straight to the decoder.
Mean IoU across classes is the standard evaluation metric.
Upsampling can be fixed, as in nearest or bilinear, or learned, as in transposed convolution.

Common questions

What is the difference between Dice and IoU?

Both measure the overlap between a predicted mask A and a true mask B. IoU is |A ∩ B| / |A ∪ B|, and Dice is 2|A ∩ B| / (|A| + |B|), so Dice = 2·IoU / (1 + IoU) and is always at least as large. Dice loss, 1 − Dice, is popular in medical imaging because small structures still count fully even when background dominates the image.

How much labeled data does a segmentation model need?

Pixel labels are slow to make, so most projects start from an encoder pretrained on ImageNet or COCO and fine-tune on a few hundred to a few thousand masks. Heavy augmentation stretches small datasets further, as in the original U-Net. Promptable models such as Segment Anything also speed up labeling by drafting masks for a human to correct.

Can transformers do image segmentation?

Yes. SegFormer pairs a hierarchical transformer encoder with a light decoder for semantic segmentation. Mask2Former treats semantic, instance and panoptic segmentation as predicting a set of masks with a transformer decoder, and one architecture handles all three.

What is the difference between segmentation and image matting?

Segmentation gives each pixel a hard label: a pixel is either in the person or not. Matting estimates a soft opacity value between 0 and 1 for every pixel, which captures hair, fur and motion blur at the edges. Portrait-mode and background-removal tools often run segmentation first, then refine the boundary with matting.

Related explainers

Object Detection — Locate and classify multiple objects in an image using bounding boxes and confidence scores
Convolutional Neural Networks — How CNNs see and understand images
Skip Connections & ResNets — Residual learning and why deeper networks work
Pooling & Downsampling — Reduce spatial dimensions while preserving the most important features in feature maps
Diffusion Models — Generate images by learning to reverse a gradual noising process
Data Augmentation — Artificially expand training data with transformed copies to reduce overfitting

References

Last updated October 25, 2026