DL0113 VLM Object Hallucination

Object hallucination is the case where a vision-language model asserts an object that is not present in the image, and it is not one bug but the sum of four largely independent failure sources stacked along the encode → project → decode pipeline. The vision tower is usually a frozen contrastively-trained ViT, so it encodes globally discriminative semantics and is measurably weak on small objects, counts, duplicates, and absence; the projector then compresses the image to a fixed budget of visual tokens (576 for CLIP ViT-L/14 at 336 px, only 32 query tokens for a BLIP-2 style Q-Former), discarding spatial detail the decoder can never recover. On the language side, the decoder was pretrained on text alone and carries a strong object co-occurrence prior, so whenever the visual evidence for a token is weak the prior decides: “dining table” pulls in “fork”, “kitchen” pulls in “refrigerator”. Instruction tuning makes this worse rather than better, because much visual SFT data is written by a text-only model from captions and bounding boxes and therefore contains ungrounded details, and the training objective rewards confident, fluent, detailed answers with no penalty term for an unsupported noun. Finally, decoding itself drifts: attention to visual tokens concentrates on a few early anchor positions and decays with sequence position, so late sentences are generated nearly blind, and once a wrong noun is emitted it conditions everything after it, producing the well-documented snowball effect.

(1) Weak Visual Features: a contrastive image-text objective optimizes for retrieval-level discrimination, not localization, so paired images differing in one small detail can receive nearly identical embeddings.
(2) Token And Resolution Budget: a fixed low-resolution grid plus projector compression means thin, small, or crowded objects arrive at the decoder as a few blurred features.
(3) Language Co-occurrence Prior: the text-pretrained decoder completes plausible scenes, and objects that frequently co-occur with the true content are the ones hallucinated first.
(4) Instruction Data Noise And Yes-Bias: SFT data generated from captions and boxes contains details never visible, and presence questions are answered affirmatively far more often than chance.
(5) Objective Mismatch: next-token cross-entropy has no grounding loss, and helpfulness-oriented preference tuning rewards verbosity, which mechanically increases the number of nouns at risk.
(6) Autoregressive Drift: visual attention mass decays over the generated sequence and hallucinated nouns become context, so error rate grows with output length rather than with model size.

Figure 1: Each stage can independently drop or overwrite visual evidence: the frozen contrastive encoder loses fine detail, the projector loses spatial resolution, the text-only prior fills the gap, and the sampling loop recycles its own mistakes.

The two benchmark families separate these causes reasonably well. CHAIR parses generated captions against a fixed object vocabulary and reports the fraction of mentioned objects that are absent, which makes drift visible because the metric worsens as captions get longer. POPE instead asks balanced yes/no presence questions and splits the negatives three ways: random objects, popular objects, and adversarial objects chosen because they co-occur most often with the ground-truth objects. The gap between the random and adversarial splits is close to a direct measurement of language-prior dominance, and the reported yes-ratios above 90% for several 2023-era models are a direct measurement of the SFT affirmation bias. A useful diagnostic in the same spirit is to score a candidate token twice, once with the image and once with the image removed, since a near-zero difference means the prior, not the pixels, chose that word.

Mathematical Formulation:

Where:

• is the token generated at step and the tokens already committed, which is why an early mistake is irreversible.
• is the sequence of projected visual tokens (576 for a ViT-L/14 336 px grid, 32 for a Q-Former) and is the text prompt.
• is the same score with the image dropped, so is the visual grounding margin; means the token was chosen by the language prior alone.
• is the score under a distorted or noised image and the contrast strength, giving the contrastive decoding adjustment used by VCD-style mitigations.
• is the set of object mentions parsed from a caption and the subset absent from the annotation, so is an instance-level hallucination rate in .
• Required condition for the margin test: both scores must be computed at the same step with identical , otherwise the two distributions are not comparable.

Figure 2: Schematic of the drift pattern reported by attention-analysis studies and by CHAIR-versus-length ablations: as generation proceeds, attention mass on visual tokens falls while the hallucinated-object rate rises, so long free-form captions hallucinate far more than short answers from the same model.