Fully Convolutional Networks for Segmentation

Standard Convolutional Neural Networks, particularly those designed for image classification tasks like VGG or ResNet, typically end with one or more fully connected layers. These layers aggregate the spatial information learned by the convolutional filters into a fixed-size vector, suitable for predicting a single class label for the entire input image. However, this aggregation process discards the precise spatial information needed for segmentation, where we require a prediction for every pixel. If you flatten the feature map from the last convolutional layer, you lose the 2D structure.

Fully Convolutional Networks (FCNs), introduced by Long, Shelhamer, and Darrell, provide an elegant solution. The fundamental idea is to replace the fully connected layers in a classification network with equivalent convolutional layers. For instance, a fully connected layer operating on a $7 \times 7 \times 512$ feature map (common before the final classification layer in VGG) can be viewed as a convolution with a $7 \times 7$ kernel applied to that feature map, producing a $1 \times 1$ output map for each filter (class). By using $1 \times 1$ convolutions in place of fully connected layers, FCNs preserve the spatial dimensions throughout the network, allowing them to output a heatmap or a dense prediction map corresponding to the input image's spatial layout.

The Encoder-Decoder Structure

FCNs typically adopt an encoder-decoder structure:

1. Encoder (Downsampling Path): This part is usually a pre-trained classification network (like VGG, ResNet) where the final fully connected layers have been removed or converted to convolutions. As data flows through the encoder, sequences of convolution and pooling operations progressively reduce the spatial resolution ($H \times W$) while increasing the feature depth ($C$). This captures increasingly complex semantic features but loses fine-grained spatial detail.
2. Decoder (Upsampling Path): To generate a segmentation map with the same resolution as the input image, the network needs to upsample the low-resolution, high-depth feature maps from the encoder. The primary mechanism for this in FCNs is the transpose convolution (sometimes referred to as deconvolution). A transpose convolution operation essentially reverses the spatial transformation of a regular convolution, increasing the height and width of the feature map while potentially reducing its depth. It's a learnable upsampling layer.

Recovering Detail with Skip Connections

A significant challenge with the basic encoder-decoder structure is that the deep, low-resolution feature maps capture semantic context but lack precise localization information lost during pooling. Upsampling these coarse maps alone often results in blurry or poorly defined segmentation boundaries.

FCNs address this by incorporating skip connections. These connections bridge feature maps from the earlier, higher-resolution layers in the encoder directly to corresponding layers in the decoder. The decoder then combines the coarse, semantic information from the deep layers with the fine-grained, spatial detail from the shallower layers.

For example, the output from the deep encoder layers might be upsampled by a factor of 2. This upsampled map is then element-wise summed with the feature map from an earlier encoder layer that has the same spatial dimensions. This combined map is then further upsampled and potentially combined with features from even earlier layers.

Simplified FCN architecture showing the encoder reducing resolution, the decoder increasing resolution via upsampling, and skip connections combining features from different stages. N represents the number of segmentation classes.

FCN Variants: FCN-32s, FCN-16s, FCN-8s

The original FCN paper proposed several variants based on the stride of the final prediction layer and the number of skip connections used:

• FCN-32s: This is the simplest version. It takes the output of the final pooling layer (which is downsampled by a factor of 32 compared to the input), applies a $1 \times 1$ convolution to predict class scores, and then upsamples this coarse map back to the original image resolution in a single step using a transpose convolution with a stride of 32. No skip connections are used. The results are typically very coarse.
• FCN-16s: This version improves on FCN-32s. It upsamples the stride-32 prediction map by a factor of 2 and combines it (element-wise sum) with the predictions made from the preceding pooling layer (pool4, stride 16). This combined map is then upsampled by a factor of 16 to reach the final resolution.
• FCN-8s: This further refines the process. It combines the stride-32 prediction (upsampled by 2) with the stride-16 prediction (pool4), upsamples the result by 2, and then combines this with predictions made from an even earlier layer (pool3, stride 8). Finally, this combined map is upsampled by a factor of 8.

Each successive variant (32s -> 16s -> 8s) incorporates more fine-grained detail from earlier layers, leading to progressively sharper and more accurate segmentation boundaries.

Training FCNs

FCNs are typically trained end-to-end using a pixel-wise loss function. The most common choice is the average cross-entropy loss calculated independently for each pixel. For a single pixel $i$, the loss is $L_i = -\sum_{c=1}^{N_{classes}} y_{i,c} \log(p_{i,c})$, where $y_{i,c}$ is 1 if the true class of pixel $i$ is $c$ and 0 otherwise, and $p_{i,c}$ is the predicted probability (usually after a softmax activation) that pixel $i$ belongs to class $c$. The total loss for the image is the average of $L_i$ over all pixels.

Significance and Limitations

FCNs represented a major step forward for semantic segmentation with deep learning. They demonstrated that convolutional networks could be trained end-to-end for dense prediction tasks, leveraging pre-trained classification networks effectively. They also introduced the powerful concepts of converting fully connected layers to convolutions and using skip connections to fuse multi-level features.

However, FCNs still have limitations:

• Coarse Boundaries: Even FCN-8s can produce segmentation masks that aren't perfectly aligned with object boundaries, partly due to the fixed nature of pooling and upsampling operations.
• Fixed Upsampling: Using simple bilinear interpolation or transpose convolution for upsampling doesn't always capture complex boundary details effectively.
• No Instance Awareness: FCNs perform semantic segmentation. They classify pixels by category ('car', 'person') but don't distinguish between different instances of the same category ('car 1', 'car 2').

These limitations motivated the development of subsequent architectures like U-Net (which refines the symmetric encoder-decoder structure with more extensive skip connections) and DeepLab (which introduces dilated convolutions to manage spatial resolution differently), as well as instance segmentation methods like Mask R-CNN, which we will explore later in this chapter. Nonetheless, understanding FCNs is foundational for grasping the core principles behind modern image segmentation networks.