通过检测模型中间层波动，仅在可疑步骤干预，有效减少视觉语言模型幻觉。
HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models
- 基于中间层令牌偏好波动识别幻觉风险点，触发校准。
- 仅在不稳步骤引入轻量探针对比，抑制错误生成。
- 无需训练，保持精度同时显著降低幻觉率，适合部署优化。
大型视觉语言模型（LVLM）虽具备强大多模态能力，但仍受视觉定位不稳定和过度依赖语言先验的影响而产生幻觉。现有无训练解码方法通常在每一步都进行校准，增加不必要的计算量且可能干扰稳定预测。本文提出一种名为‘犹豫触发差异校准’（HTDC）的无训练解码框架，通过识别层间犹豫信号——即中间层中令牌偏好波动——来判断视觉接地是否不稳定。当检测到犹豫时，HTDC激活校准机制，通过两个轻量探针（视觉归零探针与语义归零探针）对比完整分支，抑制幻觉倾向的候选输出，而在稳定步骤则维持标准全分支推理。在多个典型幻觉评测基准上的实验表明，HTDC能持续降低幻觉率，同时保持优异的任务准确率，在效果与计算开销之间取得良好平衡。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) achieve strong multimodal performance, but still suffer from hallucinations caused by unstable visual grounding and over-reliance on language priors. Existing training-free decoding methods typically apply calibration at every decoding step, introducing unnecessary computation and potentially disrupting stable predictions. We address this problem by identifying layer-wise hesitation, a simple signal of grounding instability reflected by fluctuations in token preference across intermediate layers. Based on this observation, we propose Hesitation-Triggered Differential Calibration (HTDC), a training-free decoding framework that preserves standard full-branch inference and activates calibration only at hesitation-prone steps. When triggered, HTDC contrasts the full branch with two lightweight probes, a visual-nullification probe and a semantic-nullification probe, to suppress hallucination-prone candidates while avoiding unnecessary intervention on stable steps. Experiments on representative hallucination benchmarks show that HTDC consistently reduces hallucinations while maintaining strong task accuracy, achieving a favorable trade-off between effectiveness and computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。