arXiv:2604.25642cs.CVcs.AI2026-04中稿 · CVPR被引 1

在预填充阶段干预,从源头减少视觉语言模型幻觉。

Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models

论文配图:Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models
图 1 · 摘自论文原文
  • 在预填充阶段一次性干预,修正初始键值缓存
  • 显著降低幻觉率,跨模型/解码策略通用
  • 可与现有方法插件式结合,适合追求可靠生成的场景

大型视觉语言模型(LVLM)在图文理解上取得显著进展,但其可靠性受幻觉问题严重制约,即生成与事实不符或不一致的内容。尽管近期研究通过引导向量在解码阶段缓解幻觉,但存在一个关键挑战:反而加剧了残余幻觉的严重性。我们归因于其仅关注解码阶段,错误在此自回归累积并逐步恶化后续输出。为此,我们提出预填充时间干预(PTI),一种新型引导范式,仅在预填充阶段介入一次,优化初始键值(KV)缓存以防止错误积累。具体地,PTI具备模态感知能力,为视觉和文本表示分别设计不同方向:将键导向视觉相关物体,值用于过滤背景噪声,从而在源头纠正易产生幻觉的表示。大量实验表明,PTI在缓解幻觉方面表现优异,且在多种解码策略、LVLM模型和基准测试中具备良好泛化性。此外,PTI与现有解码阶段方法正交,支持即插即用集成,并进一步提升性能。代码已公开:https://github.com/huaiyi66/PTI。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual-textual understanding, yet their reliability is critically undermined by hallucinations, i.e., the generation of factually incorrect or inconsistent responses. While recent studies using steering vectors demonstrated promise in reducing hallucinations, a notable challenge remains: they inadvertently amplify the severity of residual hallucinations. We attribute this to their exclusive focus on the decoding stage, where errors accumulate autoregressively and progressively worsen subsequent hallucinatory outputs. To address this, we propose Prefill-Time Intervention (PTI), a novel steering paradigm that intervenes only once during the prefill stage, enhancing the initial Key-Value (KV) cache before error accumulation occurs. Specifically, PTI is modality-aware, deriving distinct directions for visual and textual representations. This intervention is decoupled to steer keys toward visually-grounded objects and values to filter background noise, correcting hallucination-prone representations at their source. Extensive experiments demonstrate PTI's significant performance in mitigating hallucinations and its generalizability across diverse decoding strategies, LVLMs, and benchmarks. Moreover, PTI is orthogonal to existing decoding-stage methods, enabling plug-and-play integration and further boosting performance. Code is available at: https://github.com/huaiyi66/PTI.

视觉语言模型幻觉抑制预填充干预

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。