通过重分配注意力缓解多模态模型幻觉问题。
Reallocating Attention Across Layers to Reduce Multimodal Hallucination
- 识别感知与推理类注意力头并动态调整其层间贡献。
- 在5个基准上平均提升4.2%推理一致性，仅增加1%计算量。
- 无需训练或修改结构，适合部署在现有模型中。
多模态大推理模型常因视觉信息不足及感知与推理过程间的注意力分配失衡而产生幻觉。基于近期可解释性研究发现的分层注意力阶段化特征，我们分析了这种功能错位导致的两种互补性失效模式：浅层感知偏差与深层推理漂移。为此，提出一种轻量级、免训练插件——功能头识别与类别条件重缩放，可识别感知与推理导向的注意力头，并自适应地重新平衡其层间贡献。该方法在不重新训练或修改架构的前提下，提升了推理一致性与视觉忠实度。在三个代表性多模态推理模型和五个多模态推理基准上的评估显示，平均提升4.2个百分点，额外计算量低于1%，基线延迟增加仅9%。本研究还为调控跨层功能动态提供了可解释视角，有助于增强多模态推理的可靠性。
原文摘要 · Abstract (English)
Multimodal large reasoning models (MLRMs) often suffer from hallucinations that stem not only from insufficient visual grounding but also from imbalanced allocation between perception and reasoning processes. Building upon recent interpretability findings suggesting a staged division of attention across layers, we analyze how this functional misalignment leads to two complementary failure modes: perceptual bias in shallow layers and reasoning drift in deeper layers. To alleviate these issues, we propose Functional Head Identification and Class-Conditioned Rescaling , a lightweight, training-free plugin that identifies perception- and reasoning-oriented heads and adaptively rebalances their layerwise contributions. Our method improves reasoning consistency and visual faithfulness without retraining or any architectural modification. Evaluations across three representative MLRMs and five multimodal reasoning benchmarks show an average 4.2% point gain, with less than 1% additional computation and only 9% baseline latency. Beyond empirical improvements, our study provides an interpretable perspective on regulating cross-layer functional dynamics to enhance the reliability of multimodal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。