让视觉模型公平关注每个物体,减少幻觉并提升生成准确性。
See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment

- 通过均衡注意力分配,抑制对显著区域的过度关注。
- 在CHAIR和POPE数据集上显著降低物体幻觉率。
- 无需训练修改,适用于各类多模态大模型,适合追求生成可靠性的研究者。
多模态大语言模型常因解码时注意力过度集中于视觉显著或高频内容而产生不存在的物体幻觉。我们发现,注意力分配不均是幻觉的根本原因:当稀有、微小或情境边缘的物体获得不足关注时,模型无法充分基于完整视觉场景生成内容。为此,我们提出DOP-OBC——一种无需训练、与架构无关的解码策略,核心思想是确保图像中每个物体无论大小、频率或显著性都应有平等的表征机会。该方法引入两个互补的物体感知信号:主导物体惩罚(DOP)软性抑制对显著区域的注意力过载,异常对象增强系数(OBC)则放大对罕见但高置信度检测到的物体的关注。这些信号以每行logit调制形式注入因果注意力掩码,无需权重更新且保持自回归解码特性。跨图像与视频多模态模型的大量实验表明,DOP-OBC在CHAIR与POPE基准上持续降低物体幻觉,并在GPT-4o评估中提升描述的正确性、一致性、细节、上下文及时间维度表现。结果证明,注意力公平不仅是设计原则,更是实现更忠实多模态生成的有效路径。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) frequently hallucinate objects that are absent from the visual input, often because attention during decoding is disproportionately drawn to visually dominant or frequently occurring content. We observe that this inequity in attention allocation is a root cause of object hallucination: when rare, small, or contextually peripheral objects receive insufficient attention, the model fails to ground its generation in the full visual scene. We argue that every object in an image, regardless of its size, frequency or visual salience, deserves equal representational opportunity during decoding. To this end, we propose DOP-OBC, a training-free and architecture-agnostic decoding strategy built on the principle of equitable attention. Two complementary object-aware signals work in tandem: a Dominant Object Penalty (DOP) that softly suppresses attention over-concentration on visually dominant regions, and an Outlier Boost Coefficient (OBC) that amplifies attention toward rare yet confidently detected objects. These signals are injected as per-row logit modulations within the causal attention mask, requiring no weight updates and preserving autoregressive decoding properties. Extensive experiments across image and video MLLMs demonstrate consistent reductions in object hallucination on CHAIR and POPE benchmarks, alongside improvements in GPT-4o assessed captioning quality across correctness, consistency, detail, context and temporal dimensions. DOP-OBC establishes that fairness in attention allocation is not merely a design principle but a practical and effective path toward more faithful multimodal generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。