DL0183 Anchor-Free vs Anchor-Based Detection

The whole difference is what the regression head is measured against. An anchor-based head attaches prior boxes to every feature-map location and predicts four normalized deltas per prior, so a predicted box only exists relative to a tiled prior whose scales and aspect ratios were chosen in advance from dataset statistics. An anchor-free head has no prior: FCOS predicts the four positive distances from a location to the sides of the object it belongs to, while CornerNet and CenterNet predict heatmap peaks (two corners, or one center) plus a small size and offset regression read off at the peak. Because the reference disappears, the matching rule must change too, so IoU thresholds against priors are replaced by spatial containment plus a per-level size range, and modern detectors replace both with prediction-aware dynamic assignment. Training stability shifts rather than simply improving: anchor-free heads drop the unbounded log-space targets and the anchor hyperparameters, but they need centerness or IoU-aware quality weighting, per-level normalization of the distance targets, and an explicit tie-break for locations that fall inside two objects.

(1) Reference Frame: Anchor-based regresses log-space deltas against a discrete prior set, while anchor-free regresses geometry directly in stride units from a location or a keypoint.
(2) Matching Rule: IoU thresholds are replaced by point-in-box containment, and object scale is routed by FPN level size ranges instead of by anchor size.
(3) Candidate Count: Removing priors per location cuts predictions per level by 9x, which changes the positive/negative ratio the loss has to survive.
(4) Hyperparameter Surface: Scales, aspect ratios, and two IoU thresholds collapse into one set of level ranges plus a center-sampling radius.
(5) Stability Mechanics: Bounded positive distances with a GIoU-style box loss are better conditioned than log deltas, but need per-level scaling or a learnable exponential to keep early gradients sane.
(6) Ambiguity Handling: Two overlapping objects can claim different anchors at the same location, whereas an anchor-free point must be broken by minimum-area assignment, and CenterNet simply collides when two centers land in the same output cell.

Figure 1: The same object, three parameterizations. The anchor head needs a matched prior before its four numbers mean anything, the dense point head predicts four positive distances that are meaningful on their own, and the keypoint head turns detection into peak localization plus a size read-out, so its only remaining prior is the output stride.

Mathematical Formulation:

Anchor-Free Distances And Centerness:

Where:

• is the predicted box and the matched anchor, so the four deltas are undefined without an assignment step.
• are the anchor-based targets; the logarithm makes size ratios additive and keeps targets near zero when the prior already fits, but it is unbounded on both sides for a badly matched prior.
• is the image-space center of a feature location and its level stride, typically 8 up to 128 across P3 to P7.
• are the ground-truth corners; requiring all four of is exactly the point-in-box positive test that replaces the IoU threshold.
• is centerness, multiplied into the classification score at inference so boxes regressed from near an object border are down-ranked before NMS.
• A point is routed to level only if falls inside that level’s size interval , which is the anchor-free replacement for choosing anchor scales.

Assignment is where the two families really diverge. Faster R-CNN and YOLOv3 label a prediction by measuring IoU between a fixed prior and the ground truth, which needs a positive threshold, a negative threshold, an ignore band in between, and a rescue rule so every object keeps at least its best anchor. FCOS labels by geometry alone, then narrows the positive set with center sampling (only points within a radius of a few strides from the object center), which raises average positive quality without touching IoU at all. Keypoint methods go further and effectively skip matching: CenterNet splats a Gaussian at the object center on a stride-4 heatmap and treats the single peak cell as the only positive, which is why it needs no NMS but breaks when two centers quantize to the same cell. The lineage since then is a steady handover of the assignment decision from hand-set priors to the model itself: fixed anchors → dense point containment → adaptive statistics in ATSS → prediction-aware costs in SimOTA and TOOD.

Figure 2: One box, three positive sets. IoU matching scores 9 anchors per cell and keeps a handful, so the imbalance is roughly 1:71 here; center sampling keeps 9 of 96 points with no IoU computation at all; dynamic top-k chooses a variable number of positives from the model’s own cost, which removes thresholds but makes the label set change as training proceeds.

Candidate Count On A RetinaNet-Style FPN (800 x 1024 Input):

The count explains most of the practical differences. With about 154k anchors and a few dozen positives per image, an anchor-based one-stage head lives or dies on focal loss or a mined 1:3 negative ratio, and it pays memory and time for an IoU matrix between every anchor and every ground-truth box. Dropping to 17k points removes that matrix, cuts head parameters by the same factor, and makes the box branch predict bounded, strictly positive quantities that pair naturally with an IoU or GIoU loss instead of a smooth-L1 on log deltas. The cost is that scale and ambiguity handling become explicit design choices: level size ranges decide which stride sees an object, and a point inside two boxes is assigned to the smaller-area ground truth, a rule that still degrades on heavily nested or crowded scenes.