arXiv:2601.07359cs.CVcs.AI2026-01被引 1

不训练提升视觉理解,解决多模态模型看对说错问题

Seeing Right but Saying Wrong: Inter- and Intra-Layer Refinement in MLLMs without Training

  • 通过分层对比输出概率,追踪正确答案信念演化
  • 过滤无关注意力头,提升每层注意力质量
  • 无需训练,在多个模型和数据集上稳定增益

多模态大语言模型在多种视觉-语言任务中表现出色,但其内部推理常存在关键矛盾:尽管深层可能关注正确视觉区域,最终预测仍受早期层噪声注意力误导,导致‘看对说错’。为解决此问题,我们提出无训练的双视角解码优化方法DualPD,包含两个组件:(1) 分层注意力引导的对比逻辑模块,通过比较注意力变化最大的层间输出逻辑,捕捉正确答案信念演化;(2) 头级信息过滤模块,抑制聚焦无关区域的低贡献注意力头,提升各层注意力质量。在LLaVA与Qwen-VL模型系列上,多个多模态基准测试均表明,DualPD无需训练即可持续提升准确率,验证了其有效性与泛化能力。代码将在发表后公开。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities across a variety of vision-language tasks. However, their internal reasoning often exhibits a critical inconsistency: although deeper layers may attend to the correct visual regions, final predictions are frequently misled by noisy attention from earlier layers. This results in a disconnect between what the model internally understands and what it ultimately expresses, a phenomenon we describe as seeing it right but saying it wrong. To address this issue, we propose DualPD, a dual-perspective decoding refinement strategy that enhances the visual understanding without any additional training. DualPD consists of two components. (1) The layer-wise attention-guided contrastive logits module captures how the belief in the correct answer evolves by comparing output logits between layers that exhibit the largest attention shift. (2) The head-wise information filtering module suppresses low-contribution attention heads that focus on irrelevant regions, thereby improving attention quality within each layer. Experiments conducted on both the LLaVA and Qwen-VL model families across multiple multimodal benchmarks demonstrate that DualPD consistently improves accuracy without training, confirming its effectiveness and generalizability. The code will be released upon publication.

多模态无训练注意力优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。