DL0195 Non-Max Suppression

NMS is the greedy post-processing step that turns a detector’s dense candidate set into a short, non-redundant list of boxes, and it normally runs independently per class. Boxes below a score floor are dropped first, the survivors are sorted by confidence, the highest-scoring box moves to the output list, and every remaining box whose IoU with exceeds a threshold is deleted. The loop then repeats on whatever is left: sort → pick the max → suppress → repeat. The algorithm rests on two assumptions, that confidence ranks localization quality and that a high IoU means two boxes describe the same object. Both fail in crowds, where the best box on a partially occluded neighbour can sit above against the winner and is silently erased. Soft-NMS replaces deletion with a monotonic score decay, and DIoU-NMS subtracts a normalized center distance from the IoU so that concentric duplicates are punished harder than offset neighbours.

(1) Greedy Per-Class Loop: NMS is not an optimizer, it is a sorted sweep, so the first mistake it makes is permanent because a deleted box never re-enters the pool.
(2) Two Thresholds, Not One: a score floor (often 0.001 for AP evaluation and 0.25 for a shipped product) plus the IoU threshold in the 0.5 to 0.7 range, with a top- cap in front to bound the cost.
(3) Confidence Is The Ranking Key: classification score and box quality are only loosely correlated, so NMS can keep a confident but sloppy box and delete a precise one, which is why IoU-aware or centerness-weighted scores improve NMS without touching the algorithm.
(4) Cost: an sort plus up to pairwise IoU tests, which is why production stacks cap candidates and run a batched CUDA kernel.
(5) Dominant Failure Mode: a single global must serve sparse and crowded regions of the same image, so lowering it kills recall in crowds and raising it floods sparse regions with duplicates.
(6) Alternatives: Soft-NMS (linear or Gaussian decay), DIoU-NMS (center-distance penalty), Weighted and Cluster-NMS, Matrix NMS, and NMS-free detectors that learn one-to-one assignment instead.

Figure 1: Greedy NMS on a heavily occluded pair. The 0.81 box is a genuine duplicate and should go, but the 0.72 box is the detector’s best evidence for the second person, and its IoU of 0.53 with the winner crosses the 0.5 threshold, so hard NMS erases it. What remains for that person is a poorly localized 0.55 box, and Gaussian Soft-NMS would instead keep the 0.72 box at a decayed 0.41.

Mathematical Formulation:

Where:

• is the current candidate set for one class, a box in it with score , and the current maximum-score box that is moved to the output list .
• is the IoU threshold; line 3 is hard NMS, which zeroes (deletes) any box overlapping more than .
• Line 4 is linear Soft-NMS, applied only when ; line 5 is Gaussian Soft-NMS, applied to every remaining box with no threshold at all.
• controls the decay width, typically ; a smaller makes Soft-NMS behave more like hard NMS.
• is the Euclidean distance between the two box centers and the diagonal of the smallest box enclosing both, so lies in and is scale-invariant.
• indexes the surviving candidates, and the loop terminates when is empty or every score falls under the final report floor.

Worked Decay For The Erased Box:

Hard NMS maps that box to 0, Soft-NMS maps it to 0.41, and the practical difference is whether a report threshold of 0.3 still shows the second person. This is also the clearest way to see why Soft-NMS buys roughly 1 to 2 AP on MS COCO with no retraining: average precision rewards a correctly located box even at low confidence, because a decayed detection is ranked below the confident ones and only costs precision after all the good detections are already counted. The same property is a liability in a live product, since nothing is ever deleted and the output list keeps every candidate at some non-zero score, so a final score floor and a top- cap become mandatory rather than optional.

Figure 2: Every NMS variant is just a different score multiplier as a function of IoU. Hard NMS is a discontinuous step at , which is what makes a 0.499 and a 0.501 overlap have completely different fates. Soft-NMS swaps the step for a monotonic decay, so the cliff at the threshold disappears and the ranking, rather than a hard rule, decides what appears in the final list.

DIoU-NMS attacks a different weakness: IoU alone cannot tell a concentric duplicate from a genuinely different object. Two boxes that share a center almost certainly describe the same thing, while two boxes with the same IoU but far-apart centers are much more likely to be neighbouring instances, so the criterion becomes IoU minus the squared center distance normalized by the enclosing diagonal. The catch is that same normalization. For two tall, near-identical pedestrian boxes offset by a fraction of their width, is on the order of 0.01, so DIoU-NMS behaves almost exactly like hard NMS on the case in Figure 1. It helps most where scale or center offset differs substantially, which is why it is usually reported together with the DIoU regression loss rather than as a standalone fix for crowds.

Figure 3: Both candidates have identical IoU 0.25 with the kept box, so hard NMS treats them identically. DIoU-NMS separates them: the concentric box keeps DIoU 0.25 and is deleted, while the offset box pays a 0.10 penalty and survives at 0.15 under a 0.20 threshold. Because the penalty is divided by the enclosing-box diagonal, it only bites when the center offset is large relative to the pair’s extent.

The deeper limitation is structural rather than parametric: NMS is a non-differentiable, hand-tuned rule bolted onto a learned model, so the network is never trained to produce exactly one box per object. That is what NMS-free detectors remove. DETR-style models use Hungarian one-to-one matching during training, so duplicate suppression becomes a learned property of the decoder, and YOLOv10 keeps a one-to-many head for training signal while using a one-to-one head at inference, eliminating the NMS stage and its latency variance entirely.