DL0140 Temporal Visual Grounding

Temporal visual grounding (also called temporal sentence grounding or moment retrieval) maps one natural-language query onto a single interval inside an untrimmed video, and almost every difficulty follows from one mismatch: the metric is far more precise than the signal you can afford to look at. Benchmarks report R@1 at IoU 0.7 against a human-drawn span, yet the target moment is often a few percent of a video that runs for minutes, its boundaries are genuinely ambiguous (annotators disagree by seconds about when “opens the fridge” starts), supervision is a single positive span among thousands of candidate windows, and compute forces you to subsample frames so hard that the achievable temporal resolution can be coarser than the tolerance the metric grants. Worse, the standard datasets carry strong temporal location priors, so a model that never looks at the video can score respectably and mask the fact that no grounding is happening at all. Cross-modal features add their own problem, because image-level encoders describe objects well and describe verbs, ordering, and completion poorly, which is exactly the information a query like “right after he sits down” depends on.

(1) Tolerance Scales With The Moment, Not The Video: at a correctly sized 4-second moment tolerates a rigid boundary shift of only 0.71 s, so short moments demand sub-second precision while long ones are almost free.
(2) Boundary Ambiguity Lives In The Labels: event onsets and offsets are not crisp, so inter-annotator IoU sits well below 1.0 and an L1 boundary regression fits annotation noise past a certain accuracy.
(3) Sampling Stride Trades Precision Against Cost: sampling at fps sets a hard error floor near seconds, but keeping high makes large and early cross-modal fusion costs .
(4) Features Are Weak On Temporal Semantics: frozen CLIP-style per-frame embeddings encode appearance, so verbs, order, and “before/after” relations are nearly invisible without motion-aware or video-pretrained features.
(5) Location Priors Let Blind Models Win: on Charades-STA and ActivityNet Captions the ground-truth spans cluster in predictable places, so query-only or prior-only baselines are competitive and out-of-distribution splits expose the gap.
(6) Supervision Is Sparse And Assumes One Span: a single positive interval creates extreme foreground/background imbalance, and queries matching several disjoint intervals or none at all break single-span heads outright.

Figure 1: The tolerance the metric grants is proportional to the moment duration, so a strict threshold on short moments asks for a precision that the frame sampling stride cannot deliver. Below the dotted lines, the demanded accuracy is finer than one sampled frame, and no amount of head tuning recovers it.

Mathematical Formulation:

Where:

• is the annotated span and the prediction, both in seconds; and are the 1D intersection and union, and the handles disjoint spans.
• is the ground-truth duration, the IoU threshold, and the largest rigid shift of both boundaries that still passes, obtained from .
• indexes queries and is the usual R@1, IoU=m metric, a hard indicator that is flat inside the tolerance band and gives no gradient-like signal about how close a miss was.
• normalized by video length, and the loss weights; the L1 term is scale-sensitive and the IoU term is scale-free, which is why both appear in DETR-style grounding heads.
• is the video duration, the sampling rate, and the number of frame tokens; a 20-minute video at gives , and self-attention over it costs per query.

The design space has moved through three families and is now entering a fourth: sliding-window proposal ranking (dense candidate spans scored against the query, as in 2D-TAN), proposal-free span prediction (per-frame start and end distributions, as in VSLNet), DETR-style set prediction (learnable moment queries plus Hungarian matching with an L1 and IoU loss, as in Moment-DETR and CG-DETR), and video LLMs that emit timestamps as text or as dedicated time tokens. Each family inherits the same structural problems in a different shape: proposal methods make the imbalance explicit and cap resolution at the window grid, span prediction is cheap but assumes exactly one contiguous answer, set prediction handles multiple moments and a no-moment class but needs enough data to learn the matching, and LLM-based grounders quantize time into a token vocabulary and therefore trade fine boundary precision for reasoning ability. Scaling to hour-long input is a separate axis, because early fusion concatenates query and video tokens and blows up memory with , while late fusion keeps the video encoding query-independent so features can be cached and reused across thousands of queries.

Figure 2: Each pipeline stage contributes its own failure mode: the sampling stride fixes the resolution floor, the frozen per-frame encoder discards motion and ordering, early fusion makes cost quadratic in frame count, and a single-span head cannot express multiple or absent moments. The bottom timeline shows why the threshold choice dominates reported numbers: the same prediction passes at and fails at .