通过路由门控抑制视觉语言模型幻觉，让图像说话。
Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating
- 将注意力头拆分为视觉与文本两条路径，识别冲突
- 选择性压制主导的文本路径，保留图像路径
- 无需训练，适合提升模型可靠性
大型视觉语言模型（LVLM）常产生流畅但无图像依据的幻觉内容，影响实际应用可靠性。我们发现关键问题源于路径竞争：即使视觉标记获得注意力，最终决策仍可能被文本路径主导，使解码器优先遵循语言先验而非视觉证据。为此，提出一种无需训练、决策对齐的干预方法：将每个注意力头分解为视觉路径和文本路径，利用高效的单前向/单梯度近似估算其标记级影响。该估计揭示头内路径冲突，识别出以先验主导的路径，实现仅抑制文本路径而保留视觉路径。在五个涵盖判别与生成任务的基准上，该方法持续降低幻觉相关错误，对整体多模态性能影响小，且仅带来适度推理开销。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) often hallucinate content that is fluent yet unsupported by the image, limiting their reliability in real-world deployment. We show that a key failure mode arises from route competition: even when visual tokens receive attention, the final token decision can be dominated by the textual pathway, causing the decoder to follow linguistic priors over visual evidence. To mitigate this, we propose a training-free, decision-aligned intervention that decomposes each attention head into a visual route and a text route, and estimates their token-level effects using an efficient one-forward/one-gradient approximation. These estimates reveal route conflict within heads and identify prior-dominant ones, enabling selective suppression of only the text route while keeping the visual route intact. Across five benchmarks spanning discriminative and generative settings, our method consistently reduces hallucination-related errors across models with limited impact on overall multimodal performance, while incurring a modest inference-time overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。