通过注意力空间对比引导，让视觉语言模型生成更符合图像的描述。
Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs
- 在自注意力层直接构建图像与纯文本的对比路径，实现单次前向传播纠错。
- 在CHAIR和POPE数据集上提升描述忠实度，延迟降低至多2倍。
- 无需训练、适合部署在实时视觉生成场景中，如智能助手或内容审核。
大型视觉语言模型（LVLMs）中的幻觉现象常因语言先验压倒视觉证据而产生，导致物体误识别和视觉不一致描述。本文将幻觉缓解问题建模为注意力空间的对比引导机制，推动生成结果更贴合视觉事实且语义准确。提出无需训练的单次前向传播方法——注意力空间对比引导（ACG），直接在自注意力层操作，该层正是幻觉性跨模态偏差出现的位置。ACG在一个前向传播中构建图像条件路径与近似纯文本路径，实现在输出层错误积累前高效引导。由于掩码代理可能引入近似偏差，进一步引入轻量级正交投影，抑制与纯文本路径对齐的成分，增强视觉一致性。在CHAIR和POPE数据集上的实验表明，ACG在保持生成质量的同时，优于现有无需训练基线，在忠实度上显著提升，且相比多轮对比解码方法延迟降低最多2倍。
原文摘要 · Abstract (English)
Hallucinations in large vision--language models (LVLMs) often arise when language priors dominate over visual evidence, leading to object misidentification and visually inconsistent descriptions. We address this problem by framing hallucination mitigation as contrastive guidance that steers generation toward visually grounded and semantically faithful text. We propose Attention-space Contrastive Guidance (ACG), a training-free, single-pass method that operates directly in self-attention layers, where hallucination-inducing cross-modal biases emerge. ACG constructs both image-conditioned and approximate text-only attention paths within a single forward pass, enabling efficient guidance before errors accumulate at the output layer. Because this masking-based surrogate can introduce approximation bias, we further apply a lightweight orthogonal projection that suppresses components aligned with the text-only path, yielding a more visually grounded correction. Experiments on CHAIR and POPE show that ACG improves faithfulness over existing training-free baselines while maintaining caption quality, reducing latency by up to $2\times$ compared to multi-pass contrastive decoding methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。