arXiv:2602.24041cs.CV2026-02被引 8

通过自适应强化关键视觉信息,减少多模态大模型幻觉。

Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

  • 仅强化与语义最匹配的视觉区域,避免背景干扰。
  • 无需训练,可降低幻觉率30%以上且不损失通用能力。
  • 适合追求高可靠性的视觉推理系统开发者。

多模态大语言模型(MLLMs)在视觉-语言推理中取得显著进展,但仍易产生幻觉,即生成内容偏离视觉证据。现有缓解策略或需昂贵训练监督,或引入推理延迟。近期视觉增强方法尝试通过解码时强化视觉标记来解决,但通常盲目注入所有标记,导致背景区域干扰,分散模型对关键线索的关注。为此,我们提出无需训练的自适应视觉强化(AIR)框架。AIR包含两个组件:基于原型的标记压缩将大量视觉标记浓缩为紧凑子集以抑制冗余;基于最优传输(OT)的补丁强化量化隐藏状态与补丁嵌入间的对齐度,选择最一致的补丁注入前馈层。结果表明,AIR增强模型对显著视觉信息的依赖性,有效缓解幻觉。在多个代表性MLLM上的大量实验显示,AIR显著降低幻觉率,同时保持通用能力,是构建可靠MLLM的有效方案。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.

多模态模型幻觉抑制视觉增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。