arXiv:2510.19183cs.CVcs.AI2025-10被引 3

通过自适应裁剪键值缓存,提升视觉信息注意力,减少多模态大模型幻觉。

PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning

  • 基于视觉令牌注意力不足的观察,动态裁剪键值缓存以聚焦关键视觉信息。
  • 在多个基准上显著降低幻觉率,且不增加推理开销。
  • 无需训练、通用性强,适用于多种解码策略,适合实际部署场景。

近年来,多模态大语言模型(MLLMs)取得了显著进展,但幻觉问题仍是主要挑战。现有方法或需额外训练数据,或在推理时引入外部/内部信息,均带来额外计算成本。本文观察到,MLLM中的幻觉与视觉令牌注意力分配不足密切相关:冗余视觉令牌分散了注意力,导致关键视觉线索被忽视,从而加剧幻觉。为此,我们提出无需训练的PruneHal方法,通过自适应KV缓存裁剪增强模型对关键视觉信息的关注,有效缓解幻觉。据我们所知,这是首个将令牌裁剪用于MLLM幻觉抑制的工作。该方法几乎不增加推理开销,且模型无关,可无缝集成于各类解码策略,包括专为抑制幻觉设计的方法。我们在四个主流MLLM上,使用多个常用幻觉评估基准进行测试,结果表明该方法效果显著且优越。代码将公开。

原文摘要 · Abstract (English)

While multi-modal large language models (MLLMs) have made significant progress in recent years, the issue of hallucinations remains a major challenge. To mitigate this phenomenon, existing solutions either introduce additional data for further training or incorporate external or internal information during inference. However, these approaches inevitably introduce extra computational costs. In this paper, we observe that hallucinations in MLLMs are strongly associated with insufficient attention allocated to visual tokens. In particular, the presence of redundant visual tokens disperses the model's attention, preventing it from focusing on the most informative ones. As a result, critical visual cues are often under-attended, which in turn exacerbates the occurrence of hallucinations. Building on this observation, we propose \textbf{PruneHal}, a training-free, simple yet effective method that leverages adaptive KV cache pruning to enhance the model's focus on critical visual information, thereby mitigating hallucinations. To the best of our knowledge, we are the first to apply token pruning for hallucination mitigation in MLLMs. Notably, our method don't require additional training and incurs nearly no extra inference cost. Moreover, PruneHal is model-agnostic and can be seamlessly integrated with different decoding strategies, including those specifically designed for hallucination mitigation. We evaluate PruneHal on several widely used hallucination evaluation benchmarks using four mainstream MLLMs, achieving robust and outstanding results that highlight the effectiveness and superiority of our method. Our code will be publicly available.

多模态幻觉抑制推理优化KV缓存

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。