As the chapter introduction highlighted, image segmentation pushes computer vision towards a more granular understanding than classification or object detection. Instead of assigning a single label to an image or drawing a box around an object, segmentation assigns a category to every single pixel. This dense prediction provides a precise outline of objects and regions within the scene. However, within this goal, there's a significant distinction regarding how objects are treated, leading to two main types of segmentation tasks.

Semantic Segmentation

Semantic segmentation is the task of classifying each pixel in an image into a predefined set of categories. Think of it as assigning a semantic label (like 'road', 'sky', 'person', 'car', 'building') to every pixel. The output is typically a map of the same size as the input image, where each pixel's value corresponds to its predicted class.

Consider an image containing multiple cars on a road. A semantic segmentation model would aim to label all pixels that are part of any car as 'car', all road pixels as 'road', and so on. It understands what is present at each pixel location but does not differentiate between separate instances of the same object class. All cars belong to the single semantic category 'car'.

Characteristics:

Assigns a class label to each pixel.
Does not distinguish between different objects of the same class.
Output is a dense map of class labels.

Applications: Semantic segmentation is valuable for scene understanding where the overall context and the types of regions are important. Examples include:

Autonomous driving: Identifying drivable areas (road), sidewalks, lane markings, and static obstacles.
Medical image analysis: Delineating different types of tissues, organs, or anomalies in scans like MRI or CT.
Geospatial analysis: Classifying land cover types (water, forest, urban areas) from satellite imagery.

Instance Segmentation

Instance segmentation takes the task a step further. It not only classifies each pixel but also identifies which object instance each pixel belongs to. Returning to the example of multiple cars on a road, an instance segmentation model would identify all pixels belonging to the first car as 'car_instance_1', all pixels of the second car as 'car_instance_2', and label the road pixels appropriately as 'road'.

Essentially, instance segmentation performs object detection and semantic segmentation simultaneously. It finds individual objects and provides a precise pixel-level mask for each detected instance.

Characteristics:

Assigns a class label and a unique instance ID to pixels belonging to countable objects ('things').
Distinguishes between different objects of the same class.
For background categories ('stuff' like sky, road, grass), it behaves like semantic segmentation.
Output typically involves bounding boxes (like object detection) plus a pixel mask for each detected object instance.

Applications: Instance segmentation is needed when interacting with or analyzing individual objects within a scene. Examples include:

Robotics: Enabling a robot arm to identify and grasp a specific object from a group of similar objects.
Autonomous driving: Tracking individual vehicles or pedestrians for motion prediction.
Photo/video editing: Allowing users to precisely select and manipulate specific objects.
Biological imaging: Counting and measuring individual cells or organisms.

Visualizing the Difference

The core difference lies in whether individual objects of the same class are treated as distinct entities. Semantic segmentation groups them under one class label, while instance segmentation separates them.

Comparison highlighting the different goals and outputs of semantic and instance segmentation for an image containing multiple objects of the same class.

Understanding this distinction is fundamental because the network architectures, loss functions, and evaluation metrics often differ between semantic and instance segmentation tasks. Instance segmentation is generally considered a more complex problem as it requires both correct classification and accurate delineation of potentially overlapping object instances. As we proceed through this chapter, we will examine architectures suited for both, starting with foundational methods often used for semantic segmentation and then moving towards approaches capable of instance-level predictions.