Anchor Boxes: Design and Refinement

Detecting objects requires not only classifying image regions but also precisely localizing them with bounding boxes. Early methods often relied on computationally expensive sliding window approaches. Modern detectors, particularly single-stage ones like YOLO and SSD, and the proposal stage of two-stage detectors like Faster R-CNN, employ a more efficient concept: anchor boxes (also sometimes called default boxes or priors). These predefined boxes provide reference templates that the network learns to refine.

Think of anchor boxes as a set of initial guesses for bounding boxes, strategically placed across the image at different locations and with varying sizes and shapes. Instead of predicting box coordinates from scratch, the network learns to predict:

1. An objectness score for each anchor: How likely is it that this anchor contains the center of an object?
2. Offsets: How should this anchor box be adjusted (moved, resized) to tightly fit the actual object?
3. Class probabilities: If an object is present, what class does it belong to?

These anchor boxes are typically defined relative to the cells of the output feature map of a convolutional backbone. For a feature map of size $W \times H$ with $C$ channels, each of the $W \times H$ spatial locations (cells) is associated with a set of $k$ anchor boxes. Each anchor box has a predefined scale (size) and aspect ratio (width-to-height ratio).

Anchor boxes with different scales and aspect ratios associated with a single spatial location on a feature map, projected onto the corresponding region in the input image.

Designing Anchor Box Sets

The effectiveness of an anchor-based detector heavily depends on the appropriate selection of anchor box scales and aspect ratios. These choices directly influence the model's ability to detect objects of various sizes and shapes. Common strategies include:

1. Manual Selection: Define a range of scales (e.g., increasing powers of 2) and aspect ratios (e.g., 1:1, 1:2, 2:1, 1:3, 3:1) based on general knowledge or preliminary dataset analysis. This is common in SSD and Faster R-CNN.
2. Dataset Statistics (Clustering): Analyze the dimensions of ground truth bounding boxes in the training dataset. Use a clustering algorithm, like k-means, on the box dimensions (often normalized width and height) to find the most representative shapes and sizes. This approach, popularized by YOLOv2 and later versions, aims to tailor the anchors specifically to the target dataset. The distance metric for clustering is often based on Intersection over Union (IoU) rather than Euclidean distance to better reflect bounding box similarity.

The number of anchors ($k$) per location is also a design choice. Using more anchors increases coverage and potentially recall, especially for unusually shaped objects, but also increases computational cost and the number of predictions the network must make.

Matching Anchors to Ground Truth during Training

During training, each anchor box needs to be labeled as either containing an object (positive) or background (negative). This assignment is typically done using IoU between the anchor boxes and the ground truth bounding boxes.

A common matching strategy involves:

1. Calculating the IoU between every anchor box and every ground truth box.
2. Assigning an anchor as positive if its IoU with any ground truth box exceeds a high threshold (e.g., IoU > 0.7). To ensure every ground truth box is covered, the anchor(s) with the highest IoU for each ground truth box might also be assigned as positive, even if below the high threshold.
3. Assigning an anchor as negative if its maximum IoU with all ground truth boxes is below a low threshold (e.g., IoU < 0.3).
4. Anchors falling between the low and high thresholds are typically ignored during training for the classification and regression loss calculations to avoid ambiguity.

Refining Anchors via Bounding Box Regression

For anchors matched positively to a ground truth box, the network learns to predict refinement offsets. Instead of predicting absolute coordinates ($x, y, w, h$), the network predicts four delta values ($t_x, t_y, t_w, t_h$) relative to the anchor box's properties ($a_x, a_y, a_w, a_h$).

A widely used parameterization (similar to Faster R-CNN) is:

$$t_x = (x_{gt} - a_x) / a_w$$ $$t_y = (y_{gt} - a_y) / a_h$$ $$t_w = \log(w_{gt} / a_w)$$ $$t_h = \log(h_{gt} / a_h)$$

Here, $(x_{gt}, y_{gt}, w_{gt}, h_{gt})$ are the center coordinates, width, and height of the ground truth box, and $(a_x, a_y, a_w, a_h)$ are the corresponding properties of the anchor box. The network is trained using a regression loss (like Smooth L1 loss) to minimize the difference between its predicted $t$ values and the target $t$ values calculated from the matched ground truth box.

At inference time, the network predicts $(t_x, t_y, t_w, t_h)$. These predictions are then used to transform the initial anchor box $(a_x, a_y, a_w, a_h)$ into the final predicted bounding box $(p_x, p_y, p_w, p_h)$:

$$p_x = t_x \cdot a_w + a_x$$ $$p_y = t_y \cdot a_h + a_y$$ $$p_w = \exp(t_w) \cdot a_w$$ $$p_h = \exp(t_h) \cdot a_h$$

This regression mechanism allows the network to precisely adjust the size and position of the predefined anchors to match the detected objects accurately. The final set of refined boxes, along with their confidence and class scores, are then typically processed using Non-Maximum Suppression (NMS) to eliminate redundant detections, which we discuss in the next section.

Choosing the right anchor configuration involves trade-offs between detection performance across different object scales/ratios and computational efficiency. While anchor boxes have been foundational to many successful object detectors, ongoing research also explores anchor-free methods that aim to predict object locations more directly, removing the need for predefined anchor sets and their associated hyperparameters. However, understanding anchor boxes remains important for working with many state-of-the-art detection models.