1Pooling & Downsampling 2Object Detection 3Image Segmentation 4Data Augmentation

Lensa ML

Object Detection

Step 1 of 6: Classification vs Detection

From labels to bounding boxes

Number of Objects1

Objects

Mode

Classify

Outputs

1 label

Classification assigns a single label to the entire image. It answers "what is this?" but not "where is it?"

Drag Number of Objects from 1 to 5. For each object, a detector outputs a class label, a confidence score and a box, usually four numbers such as (x_min, y_min, x_max, y_max) or a center with a width and height. The number of objects changes from image to image, so detectors predict a large, fixed set of candidate boxes and filter them afterwards, which is what the later steps cover. Segmentation goes one level further and labels every pixel. The standard benchmark is COCO, with 80 object classes.

Step 2 of 6: Anchor Boxes

Predefined reference shapes on a grid

Anchors per Cell1

Anchors / cell

Grid cells

Total anchors

With 1 square anchor per cell, only roughly square objects get a close match. Tall or wide objects may be missed.

Drag Anchors per Cell from 1 to 4. Each anchor is a reference box with a fixed position, size and aspect ratio. For every anchor, the network predicts whether it contains an object and how to adjust it. Faster R-CNN uses 9 anchors per location (3 scales × 3 aspect ratios), so a 60×40 feature map gives over 20,000 anchors. In training, an anchor counts as positive when its IoU with a ground-truth box passes a threshold, 0.7 in Faster R-CNN. Anchor-free detectors such as FCOS and CenterNet predict boxes from points instead.

Step 3 of 6: Bounding Box Regression

Refining anchor positions to match objects

Regression Offset0.00

Offset

0.00

IoU

0.000

0px

The predicted box is still close to the anchor. Larger offsets are needed to reach the ground truth.

Drag Regression Offset from 0 to 1. Detectors do not predict raw pixel coordinates. Faster R-CNN predicts t_x = (x − x_a)/w_a, t_y = (y − y_a)/h_a, t_w = log(w/w_a) and t_h = log(h/h_a), offsets scaled by the anchor's size, so the same weights work for small and large boxes. Training uses a smooth L1 loss on these offsets or an IoU-based loss such as GIoU. At offset 0 the IoU is exactly 0, because the boxes only touch, and plain IoU then gives no gradient.

Step 4 of 6: Confidence & Class Scores

Filtering detections by confidence

Confidence Threshold0.30

Kept

Removed

Precision

0.80

Recall

0.80

At threshold 0.30, weaker detections are filtered out. This balances precision (0.80) and recall (0.80).

Drag Confidence Threshold from 0 to 1 across the seven detections, which score from 0.15 to 0.95. In YOLO, a box's score is its objectness times the class probability, so a box can be sure something is there but unsure what it is. Sweeping the threshold traces a precision-recall curve, and the area under it is the average precision (AP). COCO's headline metric averages AP over 80 classes and over IoU thresholds from 0.5 to 0.95. Two of the seven boxes, Bird and Tree, are false positives, so at the default threshold of 0.30 precision is 4 of 5 kept boxes, 0.80, and recall is 4 of 5 real objects, also 0.80.

Step 5 of 6: Non-Maximum Suppression

Removing duplicate detections

IoU Threshold0.50

IoU threshold

0.50

Input boxes

Surviving

NMS keeps the highest-confidence box and removes neighbors with IoU above 0.50. Duplicates of the same object go, and separate objects stay.

Drag IoU Threshold from 0.1 to 0.9. NMS sorts boxes by confidence, keeps the top one, deletes every remaining box whose IoU with it is at or above the threshold, and repeats. Box 1 (0.95) and box 2 (0.82) have an IoU of 0.680, so box 2 is deleted at 0.5 and kept once the threshold passes 0.68. Two real objects that overlap heavily, such as people in a crowd, can be wrongly suppressed. Soft-NMS lowers their scores instead of deleting them, and DETR removes the need for NMS altogether.

Step 6 of 6: Single-Stage vs Two-Stage

YOLO vs Faster R-CNN tradeoffs

Architecture Blend (YOLO ↔ Faster R-CNN)0.00

Architecture

1-Stage

Speed (FPS)

mAP

33.0%

Single-stage detectors like YOLO predict bounding boxes and classes in one pass over the grid. This yields high speed (~50 FPS) but may sacrifice accuracy on small objects.

Drag the Architecture Blend slider from YOLO to Faster R-CNN. The FPS and mAP readouts are illustrative, not benchmark results. Faster R-CNN (Ren et al., 2015) shares one backbone between a region proposal network and a second stage that classifies and refines each proposal. YOLO (Redmon et al., 2016) treats detection as a single regression over a grid and ran at 45 frames per second. The gap has since narrowed: RetinaNet's focal loss let a single-stage detector match two-stage accuracy by down-weighting the many easy background anchors.

What is object detection?

Object detection is the computer vision task of finding every object of interest in an image and returning a bounding box, a class label and a confidence score for each one. It combines classification, which says what is in the image, with localization, which says where.

Detection runs in self-driving perception, pedestrian and traffic-sign recognition, retail shelf monitoring, medical imaging for finding nodules or lesions, and photo search. Most detectors share the same parts: a CNN or transformer backbone, a set of candidate boxes such as anchors, a head that scores and refines each candidate, and non-maximum suppression to remove duplicates.

Key ideas

A detector outputs a box, a class label and a confidence score for each object.
Anchors are reference boxes at fixed positions, sizes and aspect ratios.
The network predicts offsets from each anchor, scaled by its size, instead of raw coordinates.
IoU, the overlap area divided by the union area, measures how well two boxes match.
Non-maximum suppression keeps the highest-scoring box and removes overlapping duplicates.
Single-stage detectors are faster, and two-stage detectors refine region proposals in a second step.

Common questions

What is the difference between object detection and image segmentation?

Detection draws a rectangle around each object. Segmentation labels every pixel, so it captures exact shapes. Instance segmentation combines the two: Mask R-CNN adds a mask branch to a Faster R-CNN detector and outputs a pixel mask inside each box.

How do you train an object detector on a custom dataset?

Start from a detector pretrained on COCO and fine-tune it on your own labeled images, keeping the backbone and replacing the class head. Each image needs a box and class for every object, usually stored in COCO JSON or YOLO text format. Transfer learning lets a few hundred labeled images per class give usable results for many tasks.

Why are small objects hard to detect?

A CNN backbone downsamples the image by 16 or 32 times, so a 20×20-pixel object covers only one or two cells of the final feature map. Feature Pyramid Networks (Lin et al., 2017) fix much of this by detecting from several resolutions at once. Higher input resolution and tiling large images also help.

What is DETR?

DETR (Carion et al., 2020) is a transformer detector that predicts a fixed-size set of boxes directly. During training, Hungarian matching pairs each ground-truth object with one prediction, so the model learns to avoid duplicates. It needs no anchors and no NMS.

Related explainers

Convolutional Neural Networks — How CNNs see and understand images
Image Segmentation — Classify every pixel in an image — from semantic regions to individual object instances
Pooling & Downsampling — Reduce spatial dimensions while preserving the most important features in feature maps
Transfer Learning — Standing on the shoulders of pretrained giants
Model Evaluation & Metrics — Precision, recall, F1, ROC curves, and regression metrics
Data Augmentation — Artificially expand training data with transformed copies to reduce overfitting

References

Last updated October 25, 2026