While two-stage detectors like Faster R-CNN achieve high accuracy by first proposing regions and then classifying them, this sequential process can introduce computational overhead. Single-stage detectors streamline this process by performing localization and classification simultaneously, directly predicting bounding boxes and class probabilities from feature maps in a single pass through the network. This architectural choice generally leads to faster inference times, making them suitable for real-time applications. We'll examine two influential single-stage detectors: the Single Shot MultiBox Detector (SSD) and RetinaNet.

Single Shot MultiBox Detector (SSD)

SSD tackles the challenge of detecting objects at various scales by making predictions from multiple feature maps at different resolutions within a single forward pass of a base network (like VGG or ResNet). Earlier feature maps (closer to the input) have higher spatial resolution and capture finer details, making them suitable for detecting smaller objects. Later feature maps have lower resolution but larger receptive fields, enabling them to detect larger objects.

Architecture and Multi-Scale Feature Maps: SSD starts with a standard classification network (the backbone), truncated before the final classification layers. It then appends several auxiliary convolutional layers that progressively decrease in spatial resolution while increasing the channel depth. Unlike models that only predict from the final feature layer, SSD generates predictions from selected feature maps at various stages of this backbone and auxiliary structure.

For each selected feature map, a set of convolutional filters predicts:

Bounding Box Offsets: Refinements relative to predefined default boxes (similar to anchor boxes).
Class Confidence Scores: Probabilities for each object class (including a background class).

Default Boxes (Anchors): At each location on a selected feature map, SSD associates a set of default boxes with different aspect ratios and scales. These default boxes tile the feature map and serve as initial proposals. The network predicts offsets ( $\Delta cx, \Delta cy, \Delta w, \Delta h$ ) to adjust the position and size of these default boxes to better fit the ground truth object, along with the confidence scores for each class. The scales of default boxes are typically smaller for higher-resolution feature maps and larger for lower-resolution maps, aligning box sizes with the expected object sizes at that feature level.

Training: During training, each ground truth bounding box is matched to the default box with the highest Jaccard overlap (Intersection over Union, IoU). Default boxes with an IoU greater than a threshold (e.g., 0.5) with any ground truth box are also considered positive matches. All other default boxes are marked as negative (background). The loss function is a weighted sum of:

Localization Loss (e.g., Smooth L1): Calculated only for positive matches, penalizing errors in predicted box offsets.
Confidence Loss (e.g., Cross-Entropy): Calculated for both positive and negative matches, penalizing classification errors. Hard negative mining is often used to balance the large number of negative boxes.

SSD offers a good balance between speed and accuracy, often significantly faster than two-stage detectors. However, because it uses relatively lower-resolution feature maps for detecting larger objects and relies heavily on these pre-defined default boxes across multiple scales, it sometimes struggles with detecting very small objects accurately compared to methods that analyze finer details more thoroughly or have dedicated proposal stages.

RetinaNet: Addressing Class Imbalance with Focal Loss

A primary challenge for dense, single-stage detectors like SSD is the extreme class imbalance during training. The vast majority of default boxes or anchor locations correspond to the background class, while only a small fraction represent actual objects. Standard cross-entropy loss applied to all locations means that the easily classified background examples can collectively dominate the loss value and gradient updates, hindering the network's ability to learn effective representations for the rarer foreground object classes.

RetinaNet introduced the Focal Loss specifically to address this imbalance. It's a dynamically scaled cross-entropy loss where the scaling factor decays to zero as confidence in the correct class increases. Intuitively, Focal Loss automatically down-weights the contribution of easy examples (typically abundant background negatives) during training and focuses the model's attention on hard-to-classify examples (often foreground objects or ambiguous background patches).

The standard Cross-Entropy (CE) loss for binary classification can be written as: $CE(p, y) = \begin{cases} -\log(p) & \text{if } y=1 \\ -\log(1-p) & \text{if } y=0 \end{cases}$ Where $y \in \{0, 1\}$ is the ground truth class and $p \in [0, 1]$ is the model's estimated probability for the class $y=1$ . We can rewrite this more compactly as $CE(p_t) = -\log(p_t)$ , where $p_t$ is defined as: $p_t = \begin{cases} p & \text{if } y=1 \\ 1-p & \text{if } y=0 \end{cases}$ $p_t$ represents the probability of the ground truth class.

The Focal Loss adds a modulating factor $(1 - p_t)^\gamma$ to the standard cross-entropy loss, with a tunable focusing parameter $\gamma \ge 0$ : $FL(p_t) = -(1 - p_t)^\gamma \log(p_t)$ Optionally, an $\alpha$ -balancing parameter can also be added: $FL(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t)$ Where $\alpha_t$ is $\alpha$ for class 1 and $1-\alpha$ for class 0.

Properties of Focal Loss:

Down-weighting Easy Examples: When an example is easily classified ( $p_t \rightarrow 1$ ), the modulating factor $(1 - p_t)^\gamma$ approaches 0, reducing the loss contribution for that example. Easy background examples with high confidence ( $p \rightarrow 0$ , so $p_t \rightarrow 1$ ) have their loss significantly reduced.
Focusing on Hard Examples: When an example is misclassified ( $p_t$ is small), the modulating factor approaches 1, and the loss is similar to standard cross-entropy. The relative contribution of these hard examples increases.
The Focusing Parameter $\gamma$ : Increasing $\gamma$ increases the effect of the modulating factor. For $\gamma = 0$ , Focal Loss is equivalent to standard Cross-Entropy. Empirically, $\gamma=2$ was found to work well in the original RetinaNet paper.

The plot illustrates how Focal Loss (FL) with $\gamma > 0$ reduces the loss for well-classified examples (high $p_t$ ) compared to standard Cross-Entropy (CE, equivalent to $\gamma = 0$ ). Higher values of $\gamma$ increase this effect, focusing training on harder examples where $p_t$ is low.

RetinaNet Architecture: While Focal Loss is the main contribution, the RetinaNet detector itself typically employs a Feature Pyramid Network (FPN) built on top of a backbone like ResNet. FPN generates a multi-scale feature pyramid with rich semantics at all levels, improving the detection of objects across a wide range of scales, complementing the effect of Focal Loss. Anchor boxes are applied to each level of the feature pyramid for prediction.

By effectively managing class imbalance with Focal Loss, RetinaNet demonstrated that single-stage detectors could achieve accuracy comparable to or even exceeding popular two-stage detectors like Faster R-CNN, while maintaining higher speeds.

Summary of SSD and RetinaNet

Both SSD and RetinaNet exemplify the single-stage approach, directly predicting boxes and classes without a dedicated region proposal step.

SSD introduced the idea of using multiple feature maps within a single network pass for multi-scale detection, offering a significant speed advantage. Its primary limitation can be accuracy, particularly for small objects.
RetinaNet identified and solved the critical class imbalance problem inherent in dense single-stage detectors via the novel Focal Loss. Combined with an FPN backbone, it closed the accuracy gap with two-stage methods while retaining the efficiency benefits of single-stage designs.

Choosing between these (and other detectors like YOLO) often involves considering the specific application's requirements regarding speed, accuracy, object size distribution, and scene complexity. RetinaNet generally provides higher accuracy than SSD, especially in challenging scenarios, thanks to Focal Loss and often the use of FPN, while SSD might be preferred if maximum speed is the absolute priority and its accuracy trade-offs are acceptable.