arXiv:2605.04641cs.CV2026-05被引 1

用图文描述引导注意力,有效减少大模型幻觉。

CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering

论文配图:CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering
图 1 · 摘自论文原文
  • 通过图文问答激活视觉注意力,引导模型关注真实图像内容。
  • 在5个模型、5个数据集上平均降低6.03%幻觉率。
  • 无需训练和额外计算,适合快速部署到现有模型。

尽管大型视觉语言模型(LVLMs)在下游任务中表现优异,但常产生与视觉信息不符的内容,导致对象幻觉。现有方法多依赖昂贵的人工标注或训练成本,或解码策略,显著增加推理时间。本文观察到,当回答图文描述问题时,LVLM对视觉信息的注意力明显增强。受此启发,提出无需训练、可即插即用的“图文引导视觉注意力调制”(CAST)方法,利用图文问答对应的注意力激活模式,提升模型视觉感知能力。具体而言,通过探测技术识别对图文查询敏感的注意力头,并估计其输出的优化调制方向,从而强化模型细粒度视觉理解,有效缓解对象幻觉。CAST在五个主流LVLM和五个基准测试(涵盖判别与生成任务)上平均降低6.03%幻觉率,性能达到当前最优,且推理开销极低,同时保持其他基础能力。

原文摘要 · Abstract (English)

Although Large Vision-Language Models (LVLMs) have demonstrated remarkable performance on downstream tasks, they frequently produce contents that deviate from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or decoding strategies which significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly enhanced when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-guided Visual Attention Steering (CAST), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern corresponding to caption queries to enhance LVLMs' visual perception capability. Specifically, we use probing techniques to identify attention heads that are highly sensitive to caption queries and estimate optimized steering directions for their outputs. This steering strengthens LVLM's fine-grained visual perception capabilities, thereby effectively mitigating object hallucination. CAST reduced object hallucination by an average of 6.03% across five widely used LVLMs and five benchmarks including both discriminative and generative tasks, demonstrating state-of-the-art performance while adding little inference cost and preserving other foundational capabilities.

视觉语言模型幻觉抑制注意力调制无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。