arXiv:2508.01653cs.CVcs.AI2025-08被引 2

用视觉语义地图优化大模型生成,减少幻觉。

MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processing

  • 将模型隐藏状态视为2D语义地图,全局捕捉事实信息。
  • 通过跨层注意力逐步优化标记表示,提升事实一致性。
  • 无需训练,适配多基准测试,适合追求准确性的研究者。

大型视觉语言模型(LVLMs)在多模态任务中表现优异,但仍存在幻觉问题——生成语法正确但与视觉输入不符的内容。本文提出一种新的地图级视角,将模型隐藏状态视为二维语义地图。我们发现,真实信息广泛分布于该地图中,超出现有方法(如对比解码、逐层一致性)关注的局部区域。基于此,提出无需训练的解码方法:地图级注意力处理(MAP)。该方法采用分层交叉注意力,在每层解码中融合跨层与层内信息,逐步优化标记表示;并引入全局-局部逻辑融合机制,结合全局注意力前后得到的预测结果,进一步提升准确性。在POPE、MME和MMHal-Bench等基准上,该方法持续提升模型真实性与性能,验证了地图级解码策略的有效性。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have achieved impressive performance in multimodal tasks, but they still suffer from hallucinations, i.e., generating content that is grammatically accurate but inconsistent with visual inputs. In this work, we introduce a novel map-level perspective to mitigate hallucinations in LVLMs, interpreting the hidden states of the model as a 2D semantic map. We observe that factual information is widely distributed across this map, extending beyond the localized inter- or intra-layer regions targeted by most existing methods (e.g., contrastive decoding and layer-wise consistency). Building on this insight, we propose Map-Level Attention Processing (MAP), a training-free decoding method that effectively leverages factual information through attention-based map-level operations to improve factual consistency. Specifically, we employ Layer-Wise Criss-Cross Attention to progressively refine token representations at each decoding layer by aggregating tokens from both inter- and intra-layer dimensions. Additionally, a Global-Local Logit Fusion mechanism combines logits obtained before and after global attention to further refine predictions and improve accuracy. Our method consistently improves the truthfulness and performance of LVLMs across benchmarks, such as POPE, MME, and MMHal-Bench, demonstrating the potential of the map-level decoding strategy.

视觉语言模型幻觉抑制注意力机制解码优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。