COMP90086 Computer Vision · Workshop 12

Object Detection: Region-Based CNNs, YOLO, DETR and Mean Average Precision

A primer on lectures 21 and 22, a walkthrough of the Workshop 12 notebook with the real detections on its COCO images, and the four exercises.

ILecture primerFrom sliding windows to Faster R-CNN, YOLO and DETR; NMS, precision–recall and mAP.11 interactive figures IINotebook walkthroughThe SSD MobileNet detector on 25 COCO images, evaluated as FiftyOne does it.live evaluation IIIExercises 1–4Four detectors compared, the score threshold, modified inputs, and mAP by hand.guided activities +Reference materialCheat notes, changes to the notebook code, a summary table and references.for revision
Live computed in your browser from the stored detections Precomputed run offline with TensorFlow Illustrative a small example built to show an idea
Colours: ground truth prediction · true positive false positive missed object (false negative)
Part I · Lectures 21 and 22

Lecture primer

Object detection answers two questions at once: what is in the image, and where. Each answer is a set of labelled boxes with confidence scores.

Two-stage detectors
Propose regions, then classify each one: R-CNN, Fast R-CNN, Faster R-CNN (A3–A6).
Single-pass detectors
Predict boxes and classes for the whole image in one network pass: YOLO, DETR (A7–A8). SSD and CenterNet in the notebook belong here too.
Evaluation
Remove duplicates with NMS, match predictions to ground truth by IoU, then summarise with AP and mAP (A9–A11).

Further viewing. Justin Johnson's recorded lectures cover the same material with more architecture detail: Lecture 15, Object Detection,1 and Lecture 16, Detection and Segmentation.2 Several diagrams below are redrawn from Lecture 15.

A1 · Lecture 21, slides 6–12 and 17

Detection as classification of image windows

Classification asks "is this a giraffe?". Detection asks "where is the giraffe?", and the standard answer is to ask the classification question for every window of the image.

A sliding window has three free parameters: its stride, its scale and its shape (aspect ratio). Objects come in unknown sizes, so a range of scales and shapes is needed, and the number of windows multiplies:

$$N_{\text{windows}} = N_{\text{positions}} \times N_{\text{scales}} \times N_{\text{shapes}}.$$
IllustrativeSliding windows over a COCO image

Halving the stride quadruples the number of positions. Every extra window is one more classification, and one more chance of a false alarm (slide 15).

A2 · Lecture 21, slides 13–14

Intersection over union

A detection is judged by how well its box overlaps the true box. Intersection over union (IoU) divides the shared area by the area covered by either box:

$$\operatorname{IoU}(A, B) = \frac{|A \cap B|}{|A \cup B|} = \frac{|A \cap B|}{|A| + |B| - |A \cap B|}.$$

It is 1 for identical boxes and 0 for disjoint ones. A detection counts as correct when its IoU with a true box reaches a threshold, usually 0.5.

LiveDrag either box, or its lower-right corner to resize it
ground truth \(A\)prediction \(B\)
Quick check · IoU on a grid

The ground truth covers cells \(x \in [0, 4]\), \(y \in [0, 4]\) (a \(4 \times 4\) square). A prediction covers \(x \in [2, 6]\), \(y \in [0, 4]\).

IoU of the two boxes:
Show solution

The boxes share a \(2 \times 4\) strip, so \(|A \cap B| = 8\) and \(|A \cup B| = 16 + 16 - 8 = 24\), giving \(\operatorname{IoU} = 8/24 = 1/3\). Shifting a box by half its width already drops it below 0.5.

A3 · Lecture 21, slides 18–32

Region proposals and R-CNN

Instead of every window, R-CNN classifies about 2000 regions of interest (RoIs) that are likely to contain objects. Selective Search proposes them by over-segmenting the image into superpixels and repeatedly merging adjacent ones by colour, texture, size and compactness.3

Each RoI is warped to \(224 \times 224\), run through a CNN, and given a class label and a box correction.4 The classifier has \(N + 1\) outputs: the \(N\) object classes and "background".

IllustrativeFour detector architectures built from the same blocks
runs once per imageruns once per region (drawn as a stack)where the regions come from
Advantage (slide 32)
Far fewer classifications than an exhaustive sliding window.
Disadvantages
Still about 2000 CNN passes per image, and an object missed by Selective Search can never be detected.
A4 · Lecture 21, slides 26–27

Bounding-box regression

A proposal is rarely exactly on the object. The network therefore predicts a transform \((t_x, t_y, t_w, t_h)\) that corrects the proposal \(p = (p_x, p_y, p_w, p_h)\), where \((p_x, p_y)\) is its centre.

Step 1 translates the centre by a fraction of the box size:

$$b_x = p_x + p_w\,t_x, \qquad b_y = p_y + p_h\,t_y.$$

Step 2 rescales the width and height:

$$b_w = p_w\,e^{t_w}, \qquad b_h = p_h\,e^{t_h}.$$
LiveProposal, transform and corrected box
proposal \(p\)corrected box \(b\)ground truth
Derivation · why the size uses an exponential

A width must stay positive, and a correction should mean the same for a large and a small box. Predicting \(t_w = \ln(b_w / p_w)\) gives both: any real \(t_w\) maps to a positive \(b_w\), and doubling the box is \(t_w = \ln 2\) whatever its size.

The training targets are therefore

$$t_x = \frac{g_x - p_x}{p_w}, \quad t_y = \frac{g_y - p_y}{p_h}, \quad t_w = \ln\frac{g_w}{p_w}, \quad t_h = \ln\frac{g_h}{p_h},$$

for a ground-truth box \(g\). "Set the regression targets" above fills them in.

A5 · Lecture 21, slides 33–48

Fast R-CNN and RoI pooling

Fast R-CNN runs the whole image through a fully convolutional backbone once, then crops each RoI from the last feature map.5 To give every RoI the same size, RoI pooling snaps the box to feature-map cells, divides it into a fixed grid, and max-pools within each grid cell.

With a \(640 \times 480\) image and a \(20 \times 15\) feature map, one feature column covers \(32 \times 32\) pixels:

$$\text{stride} = \frac{640}{20} = \frac{480}{15} = 32.$$
IllustrativeDrag the box: it snaps to feature cells and is pooled to a \(2 \times 2\) grid (slides 39–40)
\(20 \times 15\) feature map over a \(640 \times 480\) image
pooled \(2 \times 2\) output (one channel)
Top: the same box followed through the pipeline (adapted from J. Johnson, Lecture 15).1

Training uses a multi-task loss: cross-entropy for the class plus a smooth \(\ell_1\) loss on the box parameters, counted only for object classes (slide 47):

$$L = L_{\text{cls}} + L_{\text{loc}}, \qquad \operatorname{smooth}_{\ell_1}(e) = \begin{cases} 0.5\,e^2 & |e| < 1 \\ |e| - 0.5 & \text{otherwise.} \end{cases}$$
IllustrativeSmooth \(\ell_1\) against the squared and absolute errors
\(\operatorname{smooth}_{\ell_1}(e)\)\(0.5\,e^2\)\(|e|\)
Quadratic near zero, so small errors give gentle gradients; linear far away, so an outlier box cannot dominate the loss.

Fast R-CNN is about 9 times faster to train and 140 times faster at test time than R-CNN (slide 48), but its RoIs still come from Selective Search.

IllustrativeWhere the backbone ends and the per-region network begins (slides 44–46)
The backbone runs once on the whole image; the layers after RoI pooling run once per region, so they are kept light. With AlexNet the five conv layers form the backbone and the two FC layers the per-region network; with ResNet-50 the last stage runs per region. Adapted from J. Johnson, Lecture 15 (2019).1
A6 · Lecture 21, slides 49–54

Faster R-CNN and anchor boxes

Faster R-CNN learns the proposals too.6 A region proposal network (RPN) places an anchor point at every feature-map column, and each anchor point carries \(k\) boxes of fixed sizes and aspect ratios.

For each anchor box the RPN predicts an objectness score (object or background) and a box transform, exactly as in A4. The best proposals then go through the Fast R-CNN head.

$$N_{\text{anchors}} = W_{\text{feat}} \times H_{\text{feat}} \times k.$$
IllustrativeClick a feature-map column to see its anchor boxes

The RPN and the detection head share one backbone, so proposals cost almost nothing: Faster R-CNN tests about 10 times faster than Fast R-CNN (slide 54).

A7 · Lecture 22, slides 13–17

YOLO: one network pass, one output tensor

YOLO v1 skips proposals.7 It divides the image into an \(S \times S\) grid. Each cell predicts \(B\) boxes, each with five numbers \((x, y, w, h, \text{confidence})\), and one set of \(C\) class probabilities.

The whole output is a single tensor:

$$S \times S \times (5B + C) = 7 \times 7 \times (5 \cdot 2 + 20) = 7 \times 7 \times 30 .$$

The cell that contains an object's centre is responsible for detecting it.

IllustrativeThe grid cell responsible for each object
Loss (slide 16)
Sum-squared error between predicted and true maps; box coordinates weighted up, empty cells excluded from the class loss.
Limitation (slide 23)
One cell predicts only \(B\) boxes, so small objects packed together are hard; it is faster but less accurate than two-stage detectors.
A8 · Lecture 22, slides 18–23

DETR: detection as set prediction

DETR treats the output as a set of \(N\) (label, box) pairs, usually \(N = 100\).8 A CNN backbone and a transformer encoder produce image tokens. A transformer decoder turns \(N\) learned "object queries" into \(N\) predictions; queries without an object predict "no object".

Training first finds the one-to-one assignment \(\hat\sigma\) between ground truth and predictions with the lowest total matching cost, then applies the loss to those pairs:

$$\hat\sigma = \arg\min_{\sigma}\ \sum_{i=1}^{N} \mathcal{C}\big(y_i,\ \hat y_{\sigma(i)}\big).$$
IllustrativeBipartite matching on slide 21's example
Cost of a pair: \(1\) if the classes differ, plus the \(\ell_1\) box distance divided by 1000. Unmatched "no object" pairs cost 0. Click a ground-truth row, then a prediction, to make your own assignment.

Because each object is matched to exactly one prediction, DETR does not produce duplicate detections and needs no NMS (slide 23).

A9 · Lecture 22, slides 25–30

Non-maximum suppression

Detectors return many overlapping boxes for the same object. Non-maximum suppression (NMS) keeps one:

  1. Take the highest-scoring remaining box and keep it.
  2. Drop every lower-scoring box that overlaps it with IoU above a threshold, for example 0.7.
  3. Repeat with the next highest-scoring box. Usually done separately for each class.
IllustrativeNMS step by step on two zebras

A low threshold also suppresses correct boxes on objects that overlap each other (slide 30); a high threshold keeps duplicates.

A10 · Lecture 22, slides 31–41

Precision, recall and average precision

For one class, sort all detections by score. Walk down the list: a detection that matches an unused ground-truth box with IoU \(\ge 0.5\) is a true positive and uses up that box; otherwise it is a false positive. After each detection:

$$\text{precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}, \qquad \text{recall} = \frac{\text{TP}}{\text{number of ground-truth boxes}}.$$

Average precision (AP) is the area under the resulting precision–recall curve.

IllustrativeThe dog example of slides 32–41: click a detection to switch it between TP and FP
recall · precision

The slide reports dog AP \(\approx 0.86\): joining the points with straight lines gives 0.850, the interpolated rule below gives 0.865. The exact number depends on how the gaps between points are filled. COCO, and FiftyOne in the notebook, replace each precision by the best precision at any higher recall, then average it at 101 recall levels \(0, 0.01, \dots, 1\).9

Quick check · the best possible AP (slide 40)
Five detections, three ground-truth dogs. What is the largest AP any ordering can achieve?
Show solution

If the three true positives have the three highest scores, precision is 1 at recall \(1/3, 2/3, 1\), and AP = 1. The false positives come after recall 1, so they cost nothing. AP rewards ranking correct detections above wrong ones.

A11 · Lecture 22, slides 41–43

Mean average precision and COCO mAP

The mean average precision (mAP) averages AP over classes. The lecture's example (slide 42):

$$\text{mAP}@0.5 = \frac{0.65 + 0.80 + 0.86}{3} = 0.77 .$$

COCO mAP also averages over ten IoU thresholds, \(0.50, 0.55, \dots, 0.95\), so it rewards boxes that are tight, not just roughly right:

$$\text{COCO mAP} = \frac{1}{10} \sum_{t \in \{0.50, 0.55, \dots, 0.95\}} \text{mAP}@t .$$
LivemAP of the notebook's detector at each IoU threshold (25 images)
IoU threshold \(t\) · \(\text{mAP}@t\); the dashed line is their mean, the COCO mAP

One number hides a lot (slide 43): classification errors, loose boxes, misses and false alarms all lower mAP. The rest of this page looks inside it.

Part II · worksheet12_colab_new.ipynb

Notebook walkthrough

The notebook downloads a pre-trained detector from the TensorFlow 2 Model Zoo,10 runs it on 25 of the workshop's 50 COCO 2017 validation images, and evaluates the detections with FiftyOne.11

We ran the same code on a CPU with TensorFlow 2.21 and stored every detection with score at least 0.05. Every chart below is computed in your browser from those detections, with FiftyOne's matching and AP rules. The code cells are collapsed; their numbers follow the notebook.

Notebook section 1 · cells 1–8

Environment set-up

Install the TensorFlow Object Detection API's dependencies, FiftyOne and a MongoDB build for Colab, clone tensorflow/models, and let TensorFlow grow its GPU memory as needed.

Cell 2Install the libraries · show code
Cell 2Install the libraries
#@title Installation
!pip install tf_slim
!pip install pycocotools
!pip install fiftyone
!pip install fiftyone-db-ubuntu2204
!pip install googledrivedownloader
!pip install protobuf==3.20.*
Cell 3Pin MongoDB drivers that FiftyOne supports · show code
Cell 3Pin MongoDB drivers that FiftyOne supports
pip install 'pymongo<4.9' 'motor<3.6'
Cell 4Import FiftyOne · show code
Cell 4Import FiftyOne
import fiftyone as fo
import fiftyone.zoo as foz
Cell 5Imports and the TensorFlow models repository · show code
Cell 5Imports and the TensorFlow models repository
import os
import time
import json
import pathlib
import warnings
import numpy as np
import matplotlib.pyplot as plt
warnings.filterwarnings("ignore")     # Suppress warnings
import tensorflow as tf
tf.get_logger().setLevel('ERROR')     # Suppress TensorFlow logging

from PIL import Image
import googledrivedownloader as gdd

if "models" in pathlib.Path.cwd().parts:
    while "models" in pathlib.Path.cwd().parts:
        os.chdir('..')
elif not pathlib.Path('models').exists():
    !git clone --depth 1 https://github.com/tensorflow/models

os.chdir('models/research/')
Cell 7Grow GPU memory on demand · show code
Cell 7Grow GPU memory on demand
gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)
Cell 8Check the GPU · show code
Cell 8Check the GPU
#@title Check gpu
!nvidia-smi
Notebook section 2(1)–(2) · cells 9–12

The model zoo: speed against accuracy

The Model Zoo's detectors were all trained on COCO 2017, with its 80 object classes.9 Their names give the architecture, the backbone and the input size, for example ssd_mobilenet_v1_fpn_640x640.

The notebook uses SSD MobileNet V1 FPN 640×640: a single-pass SSD head12 on a MobileNet backbone13 with a feature pyramid.14

Model Zoo tableSpeed and COCO mAP of the ten suggested models (cell 9)
Faster R-CNN (two-stage)SSDCenterNetthe notebook's model
time per image (ms, lower is faster) · COCO mAP (higher is better)

    No model is both fastest and most accurate. The two most accurate, CenterNet Hourglass-104 (41.9) and Faster R-CNN Inception-ResNet V2 (37.7), use the heaviest backbones and are among the slowest: the speed–accuracy trade-off of lecture 21, slide 54.15

    Cell 10Download and extract a model from the zoo · show code
    Cell 10Download and extract a model from the zoo
    # Download and extract model
def download_model(model_name, model_date):
    base_url = 'http://download.tensorflow.org/models/object_detection/tf2/'
    model_file = model_name + '.tar.gz'
    model_dir = tf.keras.utils.get_file(fname=model_name,
                                        origin=base_url + model_date + '/' + model_file,
                                        untar=True, #Whether the file should be decompressed
                                        cache_dir="../../") #Location to store cached files

    model_dir = os.path.join(model_dir, model_name)
    return str(model_dir)
MODEL_DATE = '20200711'
MODEL_NAME = 'ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8'
PATH_TO_MODEL_DIR = download_model(MODEL_NAME, MODEL_DATE)
    Output of cell 10
    Downloading data from http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8.tar.gz
90453990/90453990 ━━━━━━━━━━━━━━━━━━━━ 5s 0us/step
    Cell 12Load the SavedModel · show code
    Cell 12Load the SavedModel
    PATH_TO_SAVED_MODEL = PATH_TO_MODEL_DIR + "/saved_model"

print('Loading model...', end='')
start_time = time.time()

# Load saved model and build the detection function
detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL)

end_time = time.time()
elapsed_time = end_time - start_time
print('Done! Took {} seconds'.format(elapsed_time))
    Output of cell 12
    Loading model...Done! Took 24.22835421562195 seconds
    Notebook section 2(3) · cells 13–25

    The test dataset

    Fifty images from COCO 2017 validation with their ground-truth boxes, 335 objects in 53 classes. FiftyOne stores a box as relative \([x, y, w, h]\): the top-left corner and size, each divided by the image width or height.

    DatasetThe 50 images and their ground truth
    objects per class, all 50 images (top 15)
    Click a box in the image to see it in the three coordinate conventions the notebook uses. Images marked with a dot are the 25 used for predictions.
    Cell 14Download the workshop's 50 images · show code
    Cell 14Download the workshop's 50 images
    gdd.download_file_from_google_drive(file_id='1PDUnoCwm2LrYRkDbhjmUEJaPK0EH-LNU',
                                    dest_path='../../datasets.zip',
                                    unzip=True)
    Cell 15Load the COCO-format ground truth into FiftyOne · show code
    Cell 15Load the COCO-format ground truth into FiftyOne
    # Load COCO formatted dataset ground_truth
coco_dataset = fo.Dataset.from_dir(
    dataset_type=fo.types.COCODetectionDataset,
    data_path='../../datasets/images/',
    labels_path='../../datasets/coco.json',
    include_id=True,
    label_field="ground_truth",
)
    Cell 17Inspect the dataset · show code
    Cell 17Inspect the dataset
    # Print some information about the dataset
print(coco_dataset)
    Output of cell 17
    Name:        2025.10.16.12.49.01.363076
Media type:  image
Num samples: 50
Persistent:  False
Tags:        []
Sample fields:
    id:                      fiftyone.core.fields.ObjectIdField
    filepath:                fiftyone.core.fields.StringField
    tags:                    fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:                fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:              fiftyone.core.fields.DateTimeField
    last_modified_at:        fiftyone.core.fields.DateTimeField
    ground_truth_detections: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    ground_truth_coco_id:    fiftyone.core.fields.IntField
    Cell 18One ground-truth detection · show code
    Cell 18One ground-truth detection
    # Print a ground truth detection
sample = coco_dataset.first()
print(sample.ground_truth_detections.detections[0])
    Output of cell 18
    <Detection: {
    'id': '68f0e9c29f048473143f7f9d',
    'attributes': {},
    'tags': [],
    'label': 'person',
    'bounding_box': [
        0.30445833333333333,
        0.46070312500000005,
        0.22441666666666665,
        0.293421875,
    ],
    'mask': None,
    'mask_path': None,
    'confidence': None,
    'index': None,
    'supercategory': 'person',
    'iscrowd': 0,
}>
    Cell 19The 80 COCO categories · show code
    Cell 19The 80 COCO categories
    # Export labels in COCO format
with open("../../datasets/coco.json") as f:
    data = json.load(f)

# Explore labels
data['categories']
    Output of cell 19
    [{'supercategory': 'person', 'id': 1, 'name': 'person'},
 {'supercategory': 'vehicle', 'id': 2, 'name': 'bicycle'},
 {'supercategory': 'vehicle', 'id': 3, 'name': 'car'},
 {'supercategory': 'vehicle', 'id': 4, 'name': 'motorcycle'},
 {'supercategory': 'vehicle', 'id': 5, 'name': 'airplane'},
 {'supercategory': 'vehicle', 'id': 6, 'name': 'bus'},
 {'supercategory': 'vehicle', 'id': 7, 'name': 'train'},
 {'supercategory': 'vehicle', 'id': 8, 'name': 'truck'},
 ...  (80 categories in all)
    Cell 21Launch the FiftyOne App · show code
    Cell 21Launch the FiftyOne App
    session = fo.launch_app(coco_dataset)
    Cell 23Print the App's URL · show code
    Cell 23Print the App's URL
    print(session.url)
    Cell 25(Optional) Load your own COCO subset · show code
    Cell 25(Optional) Load your own COCO subset
    # import fiftyone.zoo as foz

# my_dataset = foz.load_zoo_dataset(
#     "coco-2017",
#     split="validation",
#     classes=["cat", "dog"],
#     max_samples=50,
#     shuffle=True,
# )

# # Print some information about the dataset
# print(my_dataset)
    Notebook section 2(4) · cells 26–33

    Making predictions

    Cell 30 runs the detector on 25 images. The model returns up to 100 boxes per image, each as \([y_{\min}, x_{\min}, y_{\max}, x_{\max}]\) relative to the image, with a COCO class id and a score.

    The loop keeps boxes with score at least min_score_thresh = 0.3 and converts each one to FiftyOne's convention:

    $$[x, y, w, h] = [x_{\min},\ y_{\min},\ x_{\max} - x_{\min},\ y_{\max} - y_{\min}] .$$
    LiveGround truth and predictions on the 25 evaluated images
    ground truth (matched)ground truth (missed, FN)true positivefalse positive
    Cell 27Install the Object Detection API · show code
    Cell 27Install the Object Detection API
    !protoc object_detection/protos/*.proto --python_out=.
!cp object_detection/packages/tf2/setup.py .
!python -m pip install .

from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as viz_utils
    Cell 29The score threshold · show code
    Cell 29The score threshold
    min_score_thresh=.30
    Cell 30Run the detector and store the predictions · show code
    Cell 30Run the detector and store the predictions
    def load_image_into_numpy_array(path):
    """Load an image from file into a numpy array.

    Puts image into numpy array to feed into tensorflow graph.
    Note that by convention we put it into a numpy array with shape
    (height, width, channels), where channels=3 for RGB.

    Args:
      path: the file path to the image

    Returns:
      uint8 numpy array with shape (img_height, img_width, 3)
    """
    return np.array(Image.open(path).convert("RGB"))

# Exercise 3: set to True to test the detector on grayscale copies of the images
GRAYSCALE = False

# Choose a random subset of 25 samples to add predictions.
# The larger the sample you choose, the longer it will take.
predictions_view = coco_dataset.take(25, seed=51)

# Get class list
ids_to_index = {cat["id"]: idx for idx, cat in enumerate(coco_dataset.info["categories"])}
classes = [cat["name"] for cat in coco_dataset.info["categories"]]
print(len(classes))
print(classes)

# Add predictions to samples
with fo.ProgressBar() as pb:
    for sample in pb(predictions_view):

        # Load image
        image_np = load_image_into_numpy_array(sample.filepath)
        if GRAYSCALE:  # keep three channels, drop the colour
            image_np = np.stack([np.array(Image.fromarray(image_np).convert("L"))] * 3, axis=-1)

        # Convert image to tensor
        input_tensor = tf.convert_to_tensor(image_np)
        input_tensor = input_tensor[tf.newaxis, ...]  # Add batch dimension

        # Run detection
        detections = detect_fn(input_tensor)

        # Process detections
        num_detections = int(detections.pop("num_detections"))
        detections = {
            key: value[0, :num_detections].numpy()
            for key, value in detections.items()
        }
        detections["num_detections"] = num_detections

        # Convert classes to ints
        detections["detection_classes"] = detections["detection_classes"].astype(np.int64)

        result_detections = []

        for label, score, box in zip(
            detections["detection_classes"],
            detections["detection_scores"],
            detections["detection_boxes"]
        ):
            if score < min_score_thresh:
                continue

            ymin, xmin, ymax, xmax = box
            rel_box = [xmin, ymin, xmax - xmin, ymax - ymin]

            result_detections.append(
                fo.Detection(
                    label=classes[ids_to_index[label]],
                    bounding_box=rel_box,
                    confidence=float(score),
                )
            )

        # Save predictions to dataset

        sample["mytest"] = fo.Detections(detections=result_detections)
        sample.save()
    Output of cell 30
    80
['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush']
 100% |███████████████████| 25/25 [18.9s elapsed, 0s remaining, 2.4 samples/s]
    Cell 32Show the predictions in the App · show code
    Cell 32Show the predictions in the App
    session.view = predictions_view
    Notebook section 3(1) · cell 34 · Lecture 22, slides 31–43

    Metric definitions

    True positive (TP)
    A prediction matched to an unused ground-truth box of the same class with IoU \(\ge\) the threshold.
    False positive (FP)
    A prediction with no such match: wrong place, wrong class, or a duplicate.
    False negative (FN)
    A ground-truth box that no prediction matched. True negatives are not counted: every empty window would be one.

    Precision, recall and their harmonic mean \(F_1\):

    $$P = \frac{\text{TP}}{\text{TP} + \text{FP}}, \qquad R = \frac{\text{TP}}{\text{TP} + \text{FN}}, \qquad F_1 = \frac{2PR}{P + R} .$$

    Averaging over classes can be done three ways. The micro average pools the counts of all classes first; the macro average gives every class equal weight; the weighted average weights each class by its number of ground-truth objects (its support \(n_c\)):

    $$P_{\text{micro}} = \frac{\sum_c \text{TP}_c}{\sum_c (\text{TP}_c + \text{FP}_c)}, \qquad P_{\text{macro}} = \frac{1}{|C|}\sum_c P_c, \qquad P_{\text{weighted}} = \frac{\sum_c n_c P_c}{\sum_c n_c} .$$
    Quick check · micro versus macro

    Class "person": 16 TP, 17 FP. Class "car": 0 TP, 2 FP.

    1. Micro-averaged precision:
    2. Macro-averaged precision:
    Show solution

    Micro: \(16 / 35 \approx 0.457\). Macro: \((16/33 + 0)/2 \approx 0.242\). One rare class with no hits halves the macro average but barely moves the micro average.

    Notebook section 3(2) · cells 34–35

    Running the evaluation

    evaluate_detections matches predictions to ground truth class by class. Within each image and class it takes predictions from the highest score down, and gives each one the unused ground-truth box with the largest IoU, if that IoU reaches iou=0.5.

    With compute_mAP=True it also repeats the matching at the ten COCO thresholds, \(0.50, 0.55, \dots, 0.95\). So the printed mAP is the COCO mAP of A11; iou=0.5 only sets the per-object TP/FP labels.

    Cell 35Evaluate the predictions · show code
    Cell 35Evaluate the predictions
    # Evaluate the predictions in the model field of our `predictions_view`
# with respect to the objects in the `ground_truth_detections` field

results = predictions_view.evaluate_detections(
    pred_field = "mytest", #the name of the field containing the predicted
    gt_field="ground_truth_detections", #the name of the field containing the ground truth
    compute_mAP=True,
    iou=0.5, #the IoU threshold to use to determine matches
    eval_key="eval", #store per-sample and per-object results under "eval"
)

print("\nmAP: %.4f" % results.mAP())
    Output of cell 35
    Evaluating detections...
 100% |███████████████████| 25/25
Performing IoU sweep...
 100% |███████████████████| 25/25

mAP: 0.4498
    Notebook section 3(3) · cells 36–39

    Per-class results and precision–recall curves

    LiveClassification report at iou=0.5, min_score_thresh = 0.3
    LivePrecision–recall curves at IoU 0.5 (cell 39)
    recall · interpolated precision; each label gives the class's AP at IoU 0.5
    Cell 37Classification report for the 10 most common classes · show code
    Cell 37Classification report for the 10 most common classes
    # Get the 10 most common classes in the dataset by setting [:10]
# You could remove [:10] to display all classes

counts = coco_dataset.count_values("ground_truth_detections.detections.label")
classes_top10 = sorted(counts, key=counts.get, reverse=True)[:10]

# Print a classification report for the top-10 classes
results.print_report(classes=classes_top10)
    Output of cell 37
                  precision    recall  f1-score   support

        person       0.48      0.94      0.64        17
         bench       0.38      1.00      0.56         5
       handbag       0.33      0.25      0.29         4
        carrot       0.62      0.56      0.59         9
          boat       0.50      0.67      0.57         9
           car       0.00      0.00      0.00         1
  dining table       0.33      0.50      0.40         4
          book       0.12      0.33      0.18         9
      suitcase       0.40      0.50      0.44         4
      umbrella       0.60      0.60      0.60         5

     micro avg       0.39      0.64      0.48        67
     macro avg       0.38      0.53      0.43        67
  weighted avg       0.43      0.64      0.50        67
    Cell 39Plot precision-recall curves · show code
    Cell 39Plot precision-recall curves
    plot = results.plot_pr_curves(classes=["person", "bench", "car"])
plot.show()
    Notebook section 3(4) · cells 40–41

    Model mAP

    COCO mAP, 25 images
    average over 40 classes and 10 IoU thresholds
    mAP at IoU 0.5
    the PASCAL VOC-style number

    The Model Zoo lists 29.1 for this model on the 5000 COCO validation images. Our 25 images give a higher, noisier number: many classes appear once or twice, so one detection can move a class's AP from 0 to 1.

    Cell 41Print the mAP · show code
    Cell 41Print the mAP
    print(results.mAP())
    Output of cell 41
    0.4497866871508579
    Notebook section 3(5) · cells 42–45

    Sample-level analysis

    With eval_key="eval", the evaluation writes TP, FP and FN counts onto every image, and an outcome onto every box. That makes it easy to find the images where the detector struggles.

    LiveTP, FP and FN per image; click a bar to open that image above
    TPFPFN
    Cell 43The dataset now has eval fields · show code
    Cell 43The dataset now has eval fields
    # Our dataset's schema now contains `eval_*` fields
print(coco_dataset)
    Output of cell 43
    Name:        2025.10.16.12.49.01.363076
Media type:  image
Num samples: 50
Persistent:  False
Tags:        []
Sample fields:
    id:                      fiftyone.core.fields.ObjectIdField
    filepath:                fiftyone.core.fields.StringField
    tags:                    fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:                fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:              fiftyone.core.fields.DateTimeField
    last_modified_at:        fiftyone.core.fields.DateTimeField
    ground_truth_detections: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    ground_truth_coco_id:    fiftyone.core.fields.IntField
    mytest:                  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    eval_tp:                 fiftyone.core.fields.IntField
    eval_fp:                 fiftyone.core.fields.IntField
    eval_fn:                 fiftyone.core.fields.IntField
    Cell 45One prediction with its evaluation fields · show code
    Cell 45One prediction with its evaluation fields
    # Our detections have helpful evaluation data on them
sample = predictions_view.first()
det = sample.mytest.detections[0]
print(det.label, round(det.confidence, 4), det.eval, round(det.eval_iou, 4))
    Output of cell 45
    elephant 0.8228 tp 0.8586
    Part III · Notebook section 4

    Exercises

    Exercises 1–3 ask you to change the notebook and observe. The detections for those changes were run offline with the same code, so you can explore them here. Exercise 4 is a calculation by hand.

    Exercise 1 · Lecture 21, slides 44–54 · Lecture 22, slides 13–23

    Comparing detector architectures and backbones

    Problem statement

    Compare and contrast the results of Faster R-CNN, SSD and CenterNet. Also try swapping the backbone of a network (for example ResNet against MobileNet) to see if it changes the results.

    Prerequisite 1.1 · Three detector families
    Faster R-CNN
    Two stages: an RPN proposes regions from anchors, then a second head classifies and refines each one (A6).
    SSD (RetinaNet when it uses ResNet + FPN)
    One stage: anchors at several feature-map scales are classified and regressed directly, with no second look.1216
    CenterNet
    One stage, no anchors: a heatmap per class whose peaks are object centres, plus a predicted width and height at each peak.17 It extends YOLO's "heatmap per class" idea (A7).

    For a longer comparison of two-stage and single-stage detectors, see Johnson's Lectures 15 and 16.12

    Prerequisite 1.2 · The backbone

    The backbone is the feature extractor shared by every head (lecture 21, slides 44–46). MobileNet replaces full convolutions by depthwise-separable ones, which cuts computation roughly eight-fold; ResNet-50 is deeper and more accurate.1318 Swapping it while keeping the detector head isolates the backbone's effect. A5 shows where the backbone ends.

    Warm-up exercise · reading the Model Zoo table

    From cell 9: SSD MobileNet V1 FPN takes 48 ms at 29.1 mAP; SSD ResNet50 V1 FPN takes 46 ms at 34.3 mAP; Faster R-CNN Inception ResNet V2 takes 206 ms at 37.7 mAP.

    1. How many mAP points does the ResNet-50 backbone add to SSD?
    2. How many times slower is the most accurate Faster R-CNN than SSD ResNet50?
    Show solution

    ResNet-50 adds 5.2 mAP points to SSD at no measured speed cost on the zoo's TPU; on our CPU it takes 0.60 s per image against 0.54 s. The best two-stage model adds another 3.4 points for 4.5 times the time.

    Bridge to the question. The zoo's numbers are averages over 5000 images. Below, the four models run on the same 25 workshop images, so you can see where they differ object by object.

    Which model finds the most objects? Which is most precise? Does the backbone or the architecture matter more?
    PrecomputedFour detectors on the 25 evaluated images (min_score_thresh = 0.3)
    PrecomputedThe same image through each detector
    LiveAP at IoU 0.5 by class, for each detector
    Answer to Exercise 1
    • Backbone first. Swapping MobileNet for ResNet-50 under the same SSD head raises COCO mAP on the 25 images from 0.450 to 0.521 and recall from 0.65 to 0.74, for about 1.1 times the CPU time. The zoo shows the same gap (29.1 against 34.3).
    • Architecture alone does not decide. With a ResNet-50 backbone at 640×640, Faster R-CNN scores below SSD here (0.423) and on the zoo table (29.3 against 34.3). Its second look pays off with stronger backbones and larger inputs: the Inception-ResNet version reaches 37.7.
    • Different error profiles. Faster R-CNN and CenterNet keep fewer boxes above 0.3 (145 and 133 against about 200 for SSD), so they are more precise (0.54 and 0.56) but find fewer objects. SSD ResNet-50 has the best recall and the tightest boxes (mAP@0.75 of 0.573).
    • CenterNet is the fastest on our CPU (0.24 s per image): no anchors, no RoI stage, a smaller 512×512 input.
    • With 25 images the ranking is noisy: many classes have one or two objects, so a single detection swings a class's AP between 0 and 1. Conclusions need the full validation set.
    Exercise 2 · Lecture 21, slides 12 and 29 · Lecture 22, slides 31–41

    The effect of the score threshold

    Problem statement

    Try changing min_score_thresh in 2(4) Make predictions to other values between 0 and 1. How does this affect the results of detections? Why?

    Prerequisite 2.1 · A threshold picks one point on the P–R curve

    Sorting detections by score and sweeping a threshold traces the precision–recall curve (A10). A fixed threshold keeps only detections above it: it picks one operating point on that curve.

    Raising it removes low-confidence detections. Those are more often false positives, so precision tends to rise, while recall can only fall:

    $$\tau_1 < \tau_2 \ \Longrightarrow\ \text{TP}(\tau_2) \le \text{TP}(\tau_1) \ \Longrightarrow\ R(\tau_2) \le R(\tau_1) .$$
    Warm-up exercise · one class, two thresholds

    Six detections, sorted by score, with 4 ground-truth objects: scores \(0.95, 0.9, 0.7, 0.5, 0.35, 0.2\); outcomes TP, TP, FP, TP, FP, TP.

    1. Threshold 0.6: precision and recall
    2. Threshold 0.1: precision and recall
    Show solution

    At 0.6: \(P = 2/3\), \(R = 2/4 = 0.5\). At 0.1: \(P = 4/6 = 2/3\), \(R = 4/4 = 1\). Lowering the threshold doubled recall here at no loss of precision; in general it trades precision for recall.

    Bridge to the question. The same sweep on the real detections shows precision, recall and mAP move in different ways, and why.

    Move the threshold. Why does mAP fall as the threshold rises, even when precision improves?
    LivePrecision, recall, \(F_1\) and mAP against min_score_thresh (SSD MobileNet, 25 images)
    precision (micro, IoU 0.5)recall\(F_1\)COCO mAP
    The image viewer in 2(4) uses the same threshold control; set it there to see which boxes appear and disappear.
    Answer to Exercise 2
    • Higher threshold: fewer detections. Uncertain boxes, which are mostly false positives, go first, so precision rises; but some true objects are dropped too, so recall falls.
    • Lower threshold: more detections, higher recall, more false alarms and lower precision.
    • Why mAP falls as the threshold rises: AP is the area under the whole P–R curve, traced by sweeping a threshold over the scores. Applying min_score_thresh first cuts that curve off at the recall it reaches; the missing part contributes nothing. Lowering the threshold can only add points at higher recall, so mAP never decreases as the threshold falls.
    • Choosing a value depends on the application: a high threshold when false alarms are expensive, a low one when misses are. For computing mAP, keep a low threshold and let AP do the sweep.
    Exercise 3 · Lecture 21, slides 28 and 55

    Robustness to modified input images

    Problem statement

    Modify some of the input images and see if detection still works, for example by converting the images to grayscale (set GRAYSCALE = True in 2(4) Make predictions), or by splitting your own sub-dataset from COCO (2(3), optional).

    Prerequisite 3.1 · Distribution shift

    The detectors were trained on COCO's colour photos. A modified input is a test image from a different distribution: features the network learnt to rely on, such as the orange of a carrot, may be missing.

    To keep the network's input shape, a grayscale image is copied into all three channels:

    $$Y = 0.299\,R + 0.587\,G + 0.114\,B, \qquad (R', G', B') = (Y, Y, Y).$$
    Warm-up exercise · what grayscale removes
    1. A carrot-orange pixel \((255, 128, 0)\) becomes which gray level?
    2. A green pixel \((0, 255, 0)\) becomes which gray level?
    Show solution

    Orange becomes 151 and green 150. Orange and green, easy to tell apart in colour, are identical in grayscale: carrots next to broccoli lose their colour cue and must be recognised by shape and texture alone.

    Bridge to the question. Below, the four detectors see grayscale versions of the same 25 images, and the notebook's model also sees blurred ones.

    Which classes suffer most without colour? Does blur hurt more than grayscale?
    PrecomputedCOCO mAP on original and modified images (25 images)
    originalgrayscaleGaussian blur, \(\sigma = 3\) px (SSD MobileNet only)
    PrecomputedOne image, three inputs (SSD MobileNet)
    LiveChange in AP at IoU 0.5 per class, grayscale minus original (SSD MobileNet)
    Answer to Exercise 3
    • Detection still works on grayscale images, with a modest drop: COCO mAP falls by 3% (CenterNet) to 14% (SSD ResNet-50). Shape and texture carry most of the information, and COCO already contains black-and-white photographs (images 1, 9 and 46 of the dataset).
    • Classes defined by colour suffer most. For SSD MobileNet, AP at IoU 0.5 drops by 0.5 for carrot and traffic light, by 0.34 for cup and by 0.27 for broccoli.
    • Blur hurts far more: mAP falls from 0.450 to 0.239, because blur removes the edges and textures every class relies on, and small objects (bottles, balls, a traffic light) vanish first.
    • Remedies: data augmentation with the same changes (random grayscale, blur, colour jitter) during training, or fine-tuning on data from the new distribution.
    Exercise 4 · Lecture 22, slides 31–42

    Precision–recall values and mAP for one image

    Problem statement

    An object detection model was trained to detect horses. Ground-truth boxes A, B, C, D are shown in blue; detections X, Y, Z in yellow. The table gives each detection's confidence and its IoU with each ground-truth box.

    (a) Report the precision–recall values that you would use to compute mAP at IoU = 0.7 for this single image. Show your work.

    (b) How would mAP change if the IoU threshold were 0.9? Describe the result qualitatively.

    Four horses behind a fence; blue ground-truth boxes A to D and yellow detections X, Y, Z with their confidences
    Detections and their IoU with each ground-truth box
    DetectionConfidenceIoU with ground truth
    ABCD
    X0.850.51000
    Y0.900.030.820.040
    Z0.93000.360.79
    • Ground-truth horses: 4 (A, B, C, D).
    • Detections: 3 (X, Y, Z), each with a confidence score.
    • IoU threshold for a match in (a): 0.7; in (b): 0.9.
    Image and table from the notebook (cell 46); the confidence tags are added here. Blue: ground truth A–D; yellow: detections X–Z. Darker cells mean larger IoU.
    Prerequisite 4.1 · The matching rule

    Process detections from the highest confidence down. A detection is a TP if its best IoU with a still-unmatched ground-truth box reaches the threshold; that box is then used up. Recall always divides by the total number of ground-truth boxes, here 4.

    Warm-up exercise · the lecture's dog example (slides 33–39)

    Five dog detections sorted by score, 3 ground-truth dogs. The outcomes are TP, TP, FP, FP, TP.

    1. Precision and recall after the fourth detection
    2. Precision and recall after the fifth
    Show solution

    After four: \(P = 2/4 = 0.5\), \(R = 2/3\). After five: \(P = 3/5 = 0.6\), \(R = 3/3 = 1\). These are the last two points of slides 37–38.

    Bridge to the question. The horse image is the same procedure with three detections and four objects, but the IoU threshold is 0.7 instead of 0.5.

    LiveChange the IoU threshold and watch the matching
    recall · precision; shaded: AP
    Official solution, step by step
    1. Step 1 · Sort the detections by confidence
    2. Step 2 · Z matches D
    3. Step 3 · Y matches B
    4. Step 4 · X is a false positive
    5. Step 5 · The resulting AP
    6. Step 6 · (b) IoU threshold 0.9
    Reference material

    Cheat notes for the four exercises

    One card per exercise: what to remember, the formulas, the answers to the quick checks and warm-ups, and the numbers worth quoting.

    E1Comparing detectors

    Remember
    • Two-stage (Faster R-CNN): RPN proposals, then a per-RoI head. Two looks per object; pays off with strong backbones.
    • One-stage (SSD, RetinaNet, YOLO): classify anchors or grid cells in one pass. Faster.
    • CenterNet: class heatmaps, a peak per object centre, no anchors.
    • Backbone = shared feature extractor; MobileNet is light, ResNet-50 is more accurate.
    Warm-up answers
    mAP gain, ResNet-50 vs MobileNet (SSD)5.2\(34.3 - 29.1\)
    Faster R-CNN Inc-ResNet vs SSD R50 time4.5×\(206 / 46\)

    On the 25 images: SSD MobileNet V1 FPN mAP 0.450; SSD ResNet50 V1 FPN mAP 0.521; Faster R-CNN ResNet50 V1 mAP 0.423; CenterNet ResNet50 V1 FPN mAP 0.452.

    E2The score threshold

    Remember
    • A threshold picks one operating point on the P–R curve.
    • Higher threshold: precision up (usually), recall down (always non-increasing).
    • mAP integrates the whole curve, so filtering first can only lower it.
    Formulas
    $$P = \frac{\text{TP}}{\text{TP} + \text{FP}}, \quad R = \frac{\text{TP}}{\text{TP} + \text{FN}}, \quad F_1 = \frac{2PR}{P + R}$$
    Warm-up answers
    threshold 0.6\(P = 2/3,\ R = 1/2\)3 kept, 2 TP
    threshold 0.1\(P = 2/3,\ R = 1\)6 kept, 4 TP
    micro / macro precision0.457 / 0.242\(16/35\); \((16/33 + 0)/2\)

    SSD MobileNet, 25 images: threshold 0.05: mAP 0.473, threshold 0.3: mAP 0.450, threshold 0.7: mAP 0.161.

    E3Modified inputs

    Remember
    • A modified image is a distribution shift from COCO's colour photos.
    • Grayscale keeps shape and texture; blur removes edges and small objects.
    • Remedy: augment training data with the same changes.
    Formula
    $$Y = 0.299\,R + 0.587\,G + 0.114\,B$$
    Warm-up answers
    orange \((255, 128, 0)\)151\(76.2 + 75.1\)
    green \((0, 255, 0)\)150\(0.587 \times 255\)

    Grayscale: SSD MobileNet 0.450 → 0.428; SSD ResNet50 0.521 → 0.448; Faster R-CNN ResNet50 0.423 → 0.387; CenterNet ResNet50 0.452 → 0.437. Blur (SSD MobileNet): 0.239.

    E4mAP for one image

    Remember
    • Sort by confidence; match to an unused ground-truth box with IoU \(\ge\) threshold.
    • Recall divides by all ground-truth boxes (4 here).
    • AP = area under the P–R curve; with one class, mAP = AP.
    Answer (a), IoU 0.7
    Z (0.93) → D, IoU 0.79\(P = 1,\ R = 1/4\)TP
    Y (0.90) → B, IoU 0.82\(P = 1,\ R = 1/2\)TP
    X (0.85), best IoU 0.51\(P = 2/3,\ R = 1/2\)FP

    AP \(= 0.5\). (b) At IoU 0.9 nothing matches (largest IoU 0.82), so mAP drops to 0. Dog warm-up: \(P = 0.5,\ R = 2/3\), then \(P = 0.6,\ R = 1\).

    Reference material

    Changes to the notebook code

    Small edits so the code runs on current Colab and its outputs are consistent; all outputs shown on this page come from running the edited code.

    1. Cell 27 installs the Object Detection API with pip install .; current pip rejects the old --use-feature=2020-resolver flag.
    2. Cell 23 prints session.url as it is: it already ends in ?polling=true.
    3. Cell 30 converts every image to RGB before inference, and has a GRAYSCALE switch for Exercise 3, which the exercise refers to.
    4. Cell 35 passes eval_key="eval", so the per-sample fields described in section 3(5) are actually written.
    5. Cell 37 keeps the 10 most common classes, as its comment says.
    6. Cell 9's text names the model the code uses, SSD MobileNet V1 FPN 640×640.

    Our run uses TensorFlow 2.21 on a CPU. It reproduces the notebook's saved per-class report exactly except for "book" and "clock", for which the saved run recorded no detections; without those two classes our mAP is the notebook's 0.4292, and with them it is 0.4498.

    Reference material

    Summary of key results

    TopicKey formula or ruleTake-away
    Sliding windows (L21 s8–12)\(N = N_{\text{pos}} \times N_{\text{scales}} \times N_{\text{shapes}}\)exhaustive search is slow and raises false alarms
    IoU (L21 s13)\(\lvert A \cap B\rvert / \lvert A \cup B\rvert\)a match needs IoU \(\ge\) a threshold, usually 0.5
    R-CNN (L21 s19–32)≈ 2000 Selective Search RoIs, one CNN eachaccurate but ≈ 2000 CNN passes per image
    Box regression (L21 s27)\(b_x = p_x + p_w t_x,\ b_w = p_w e^{t_w}\)scale-invariant corrections to a proposal
    Fast R-CNN (L21 s34–48)one CNN pass, RoI pooling, \(L = L_{\text{cls}} + L_{\text{loc}}\)≈ 140× faster test than R-CNN
    Faster R-CNN (L21 s50–54)RPN with \(k\) anchors per positionlearned proposals, ≈ 10× faster again
    YOLO v1 (L22 s14–16)\(S \times S \times (5B + C) = 7 \times 7 \times 30\)one pass; weak on small, crowded objects
    DETR (L22 s18–22)\(\hat\sigma = \arg\min_\sigma \sum_i \mathcal C(y_i, \hat y_{\sigma(i)})\)set prediction with one-to-one matching; no NMS
    NMS (L22 s26)keep the best box, drop overlaps with IoU \(>\) thresholdremoves duplicates; can drop overlapping true objects
    AP and mAP (L22 s31–42)area under P–R; mean over classes (and IoU thresholds for COCO)rewards ranking correct detections first
    Notebook resultSSD MobileNet V1 FPN, 25 images, score \(\ge 0.3\)COCO mAP 0.450; mAP@0.5 0.594
    Reference material

    References

    1. Justin Johnson. Lecture 15: Object detection. EECS 498-007 / 598-005 Deep Learning for Computer Vision, University of Michigan, Fall 2019. Lecture recording: Michigan Online, youtube.com/watch?v=TB-fdISzpHQ. Architecture figures on this page are redrawn and adapted from this lecture, which in turn adapts figures by Ross Girshick.
    2. Justin Johnson. Lecture 16: Detection and segmentation. EECS 498-007 / 598-005 Deep Learning for Computer Vision, University of Michigan, Fall 2019. Lecture recording: Michigan Online, youtube.com/watch?v=9AyMR4IhSWQ.
    3. Jasper R. R. Uijlings, Koen E. A. van de Sande, Theo Gevers and Arnold W. M. Smeulders. Selective search for object recognition. International Journal of Computer Vision 104(2):154–171, 2013. doi:10.1007/s11263-013-0620-5
    4. Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580–587, 2014. doi:10.1109/CVPR.2014.81
    5. Ross Girshick. Fast R-CNN. In ICCV, pages 1440–1448, 2015. doi:10.1109/ICCV.2015.169
    6. Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NeurIPS, pages 91–99, 2015. arXiv:1506.01497
    7. Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi. You only look once: unified, real-time object detection. In CVPR, pages 779–788, 2016. doi:10.1109/CVPR.2016.91
    8. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213–229, 2020. doi:10.1007/978-3-030-58452-8_13
    9. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pages 740–755, 2014. doi:10.1007/978-3-319-10602-1_48
    10. TensorFlow Model Garden. TensorFlow 2 Object Detection API and Detection Model Zoo. github.com/tensorflow/models/research/object_detection
    11. Brian E. Moore and Jason J. Corso. FiftyOne. Voxel51, 2020. github.com/voxel51/fiftyone
    12. Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu and Alexander C. Berg. SSD: single shot multibox detector. In ECCV, pages 21–37, 2016. doi:10.1007/978-3-319-46448-0_2
    13. Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto and Hartwig Adam. MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861, 2017. arXiv:1704.04861
    14. Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 936–944, 2017. doi:10.1109/CVPR.2017.106
    15. Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama and Kevin Murphy. Speed/accuracy trade-offs for modern convolutional object detectors. In CVPR, pages 3296–3305, 2017. doi:10.1109/CVPR.2017.351
    16. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He and Piotr Dollár. Focal loss for dense object detection. In ICCV, pages 2999–3007, 2017. doi:10.1109/ICCV.2017.324
    17. Xingyi Zhou, Dequan Wang and Philipp Krähenbühl. Objects as points. arXiv:1904.07850, 2019. arXiv:1904.07850
    18. Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016. doi:10.1109/CVPR.2016.90

    Lecture slides: COMP90086 Computer Vision, lectures 21 (Object detection I) and 22 (Object detection II), Kris Ehinger, The University of Melbourne, Semester 2, 2026. Slide numbers are the printed slide numbers. Notebook: worksheet12_colab_new.ipynb.