DL0182 Semantic, Instance, and Panoptic Segmentation

The three tasks differ in what a label is allowed to be. Semantic segmentation assigns every pixel exactly one class out of and has no notion of object identity, so two touching cars collapse into a single connected “car” region. Instance segmentation does the opposite: it detects and masks each countable object separately with a confidence score, but it only covers thing classes, ignores amorphous stuff such as road, sky, and vegetation, and its masks may overlap each other or leave pixels uncovered. Panoptic segmentation is the union of the two: every pixel receives one pair of class plus instance id, stuff classes get a single segment with no id, things get one segment per object, and the output is a strict partition of the image with no overlaps and no unlabeled pixels except an explicit void region. Which formulation you pick follows the consumer of the output, not fashion. A planner that needs to count and track individual vehicles needs ids, whereas a free-space or sky-replacement module only needs a region mask.

(1) Label Space: semantic returns a class per pixel, instance returns a set of scored binary masks with classes, and panoptic returns a class plus an id per pixel.
(2) Things Versus Stuff: instance segmentation is defined only on countable things; semantic segmentation handles both but cannot separate two instances; panoptic handles both and separates instances.
(3) Overlap Constraint: instance masks are independent and can overlap, while panoptic forces a non-overlapping partition, which means any fusion of two heads must resolve conflicts explicitly.
(4) Metrics Differ In Kind: semantic uses mIoU, instance uses mask AP averaged over IoU thresholds and scores, and panoptic uses PQ at a single fixed matching threshold with no score sweep.
(5) Annotation Cost: stuff-only masks are cheap polygon paint, per-instance boundaries between adjacent same-class objects are the most expensive labels in the dataset.
(6) Practical Selection: use semantic for region questions, instance for counting and tracking, and panoptic when a downstream module needs one consistent scene interpretation per pixel.

Figure 1: One scene, three label spaces. The interesting pixels are the ones on the boundary between the two touching cars: semantic segmentation is structurally unable to place that boundary, instance segmentation places it but may let the two masks overlap and says nothing about road or sky, and panoptic segmentation is required to place it and to label every remaining pixel exactly once.

In practice the formulation is chosen by the consumer. Free-space and drivable-area estimation, land-cover mapping from satellite imagery, portrait or sky matting, and organ or tumor delineation are region questions, so semantic segmentation is sufficient and its cheaper labels are a real advantage. Counting, tracking, and grasping are identity questions: counting cells in a microscopy image, tracking each pedestrian across frames, or picking one item out of a bin all require per-object masks, so instance segmentation (or its video extension) is the right task. Panoptic segmentation earns its extra cost when a single downstream consumer must be handed one coherent interpretation of every pixel, which is why it is the natural output format for driving perception and robot scene understanding, where the same map must answer both “is this pixel drivable” and “which vehicle is this”. A useful diagnostic question is whether two adjacent objects of the same class must ever be told apart. If the answer is no, panoptic annotation is money spent on a distinction nobody reads.

Mathematical Formulation:

Where:

• is a pixel, its class, and its instance id, which is undefined (shared) for stuff classes and unique per object for thing classes.
• is the number of classes, split into disjoint thing and stuff subsets by the dataset definition rather than by the model.
• is the -th predicted binary mask, its class, and its confidence; the masks are independent, so they may overlap and their union need not cover the image.
• , , and in PQ count segments, not pixels, with a ground-truth segment and a prediction matched when ; that threshold makes the matching provably unique because panoptic segments cannot overlap.
• is segmentation quality, the mean IoU of matched pairs, and is recognition quality, the F1 score over segments; PQ is reported per class and then averaged, and often split into and .
• in mIoU counts pixels of class , which is why mIoU is blind to how many objects a class region contains.

Worked Example (One Image):

Four matched segments whose IoUs sum to 3.10 give an SQ of 0.775, and two false positives plus two false negatives each contribute a half count to the RQ denominator, so RQ is 0.667 and PQ lands at 0.517. The decomposition is the practically useful part: a high SQ with a low RQ means the masks are accurate but segments are being missed or hallucinated, which points at the classification and duplicate-removal path, while a high RQ with a low SQ means the right objects are found with sloppy boundaries, which points at output resolution and boundary supervision.

Figure 2: PQ is a segment-level bipartite matching followed by a product of two interpretable factors. The pair at IoU 0.38 is the detail that separates candidates who have read the metric from those who have not: it is not a weak match, it is no match at all, and it is charged once as a false positive and once as a false negative.