不训练也不修改模型,用注意力机制动态抑制视觉错误生成。
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration

- 通过区域感知的注意力重校准,动态调整错误路径权重。
- 在CHAIR、POPE等数据集上显著减少实例与句子级幻觉。
- 无需训练,适合部署于现有大模型,兼顾准确与流畅性。
大型视觉语言模型中事实性错误对象的生成(即对象幻觉)仍是持续挑战。现有方法如昂贵的数据微调、高延迟对比解码或僵化的注意力头截断,常牺牲计算效率或特征空间连续性。为此,我们提出一种无需训练的推理策略,作为区域感知的自适应加权机制,动态修正语义漂移,避免突兀的启发式截断。通过计算多个注意力头间的抗异常统计中点,建立可靠视觉表征的稳定锚点;再利用各注意力头在区域间的分歧度,动态确定干预预算,通过连续惩罚调制温和抑制诱发幻觉的注意力路径。该重校准过程有效纠正视觉-语义错位,同时完整保留生成流畅性和语言先验。在CHAIR、POPE和MME等标准多模态基准上的全面评估表明,本方法显著降低实例级与句子级幻觉,性能优于当前主流基线,验证了其高效性与算法鲁棒性。代码将公开。
原文摘要 · Abstract (English)
The generation of factually incorrect objects, commonly known as object hallucination, remains a persistent challenge in Large Vision-Language Models (LVLMs). Current approaches to address this issue - ranging from expensive data-driven fine-tuning and high-latency contrastive decoding to rigid attention head truncation - frequently compromise either computational efficiency or the continuity of the model's feature space. To overcome these limitations, we introduce a novel, training-free inference strategy that operates as a region-aware adaptive weighting mechanism to dynamically correct semantic drift without relying on abrupt heuristic truncations. By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations. We then utilize the inter-head disagreement mapped across regions to dynamically determine intervention budgets, gently suppressing hallucination-inducing attention paths through a continuous penalty modulation. This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors. Comprehensive evaluations on standard multimodal benchmarks, including CHAIR, POPE, and MME, reveal that our strategy substantially curtails both instance- and sentence-level hallucinations. The results demonstrate state-of-the-art performance against contemporary baselines, confirming our method's efficiency and algorithmic robustness. Our code will be public.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。