针对视觉语言模型幻觉问题,提出按需校准的解码方法
CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs

- 仅在低置信度时激活对比分支,减少无效计算
- 通过注意力引导局部扰动构建负样本,保留关键视觉信息
- 适合需要高可靠性推理的场景,如医疗、金融图文分析
大型视觉-语言模型虽具强大多模态推理能力,但在语言先验主导而视觉证据不足或错位时仍易产生物体幻觉。现有无训练对比解码方法通过比较原始与扰动输入的预测来缓解该问题,但或采用全局扰动破坏有效视觉信息,或每步都引入额外负分支。本文观察到幻觉风险具有瞬时性和词元特异性:生成过程中视觉注意力动态转移,部分功能词元以高置信度产出,无需对比校准。为此,提出面向大视觉-语言模型的渐进式幻觉感知解码(CHASD),一种推理时“按需校准”的框架。CHASD使用不确定性驱动的置信度门控,在下一词最大概率低于阈值时激活对比分支,并通过注意力引导对当前显著视觉词元进行局部扰动生成负样本。该设计减少不必要的负分支前向传播,同时保持高置信度步骤的原分布。在POPE、AMBER、MME、MMHal-Bench和CHAIR数据集上的实验表明,CHASD在幻觉相关指标上优于强基线方法,且具备竞争力的推理效率。
原文摘要 · Abstract (English)
Large Vision-Language Models have shown strong multimodal reasoning capabilities, yet they remain susceptible to object hallucinations when language priors dominate insufficient or misaligned visual evidence. Training-free contrastive decoding methods mitigate this issue by comparing predictions from original and perturbed visual inputs, but existing approaches either apply global perturbations that may alter useful visual evidence or invoke an additional negative branch at every decoding step. In this paper, we observe that hallucination risks are transient and token-specific: visual attention shifts across generated tokens, while some functional tokens are produced with high confidence and do not require contrastive calibration. Based on this observation, we propose Contrastive Hallucination-Aware Step-wise Decoding (CHASD) for Large Vision-Language Models, an inference-time framework for "calibration on demand". CHASD uses an uncertainty-driven confidence gate to activate the contrastive branch only when the maximum probability of the next-token is less than the threshold, and constructs the negative branch through attention-guided localized perturbations of the currently salient visual tokens. This design reduces unnecessary negative-branch forward passes while preserving the original distribution for high-confidence steps. Experiments on POPE, AMBER, MME, MMHal-Bench, and CHAIR show that CHASD improves hallucination-related metrics over strong training-free baselines with competitive inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。