通过调控注意力分布，有效减少多模态大模型幻觉问题。
ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM
- 直接调节解码时的跨模态注意力，引导模型关注文本线索。
- 在多个基准上幻觉率降低最高达38.2%，同时提升标准问答准确率。
- 无需训练、开销极小，适用于多种模型和解码方式。
多模态大语言模型常因过度依赖错误视觉线索而产生幻觉。现有方法如视觉与指令对比解码（VCD、ICD）虽能缓解此问题，但机制不透明。我们实证发现其效果与跨模态注意力重分配一致。基于此，提出注意力可调控对比解码（ASCD），在解码中直接调控注意力分数：正向增强自动挖掘的以文本为中心的注意力头（模型内稳定、跨领域鲁棒）；负向抑制实时识别的关键视觉词元。该方法运行开销极低，无需额外训练。在五种MLLM主干和三种解码策略下，ASCD在POPE、CHAIR和MMHal-Bench上将幻觉率降低最多达38.2%，同时提升MMMU、MM-VET、ScienceQA、TextVQA和GQA等标准VQA基准的准确率。这表明注意力调控是一种简单、通用且原理清晰的安全多模态生成路径。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) frequently hallucinate by over-committing to spurious visual cues. Prior remedies-Visual and Instruction Contrastive Decoding (VCD, ICD)-mitigate this issue, yet the mechanism remains opaque. We first empirically show that their improvements systematically coincide with redistributions of cross-modal attention. Building on this insight, we propose Attention-Steerable Contrastive Decoding (ASCD), which directly steers the attention scores during decoding. ASCD combines (i) positive steering, which amplifies automatically mined text-centric heads-stable within a model and robust across domains-with (ii) negative steering, which dampens on-the-fly identified critical visual tokens. The method incurs negligible runtime and memory overhead and requires no additional training. Across five MLLM backbones and three decoding schemes, ASCD reduces hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2 percent while improving accuracy on standard VQA benchmarks, including MMMU, MM-VET, ScienceQA, TextVQA, and GQA. These results position attention steering as a simple, model-agnostic, and principled route to safer, more faithful multimodal generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。