arXiv:2602.18711cs.CV2026-02被引 1

通过分层敏感度分析,精准抑制大模型幻觉而不破坏原有知识。

HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing

  • 基于分层敏感度评分,定位易出幻觉的模型层。
  • 在不加参数和延迟下,幻觉率平均降低61.8%。
  • 适合需高可靠性生成的医疗、法律等场景使用。

大型视觉语言模型(LVLMs)虽具备出色的多模态理解能力,但仍易产生物体幻觉,即描述不存在的物体或提供错误事实信息,严重制约其在真实场景中的可靠部署。尽管微调是常见缓解策略,但其高计算成本与实践难度催生了无需训练的替代方案,其中模型编辑近年成为有前景的方向。然而,盲目编辑可能破坏预训练模型中蕴含的丰富隐式知识,引发根本问题:每层应进行多大干预以抑制幻觉并保留知识?为此,我们系统分析了基于Qwen、LLaMA和Vicuna三大主流语言模型骨干的LVLM解码器,揭示各层对幻觉的敏感性存在显著差异。在此基础上,提出幻觉不敏感度评分(HIS),量化各层对幻觉的敏感程度,指导针对性干预。基于HIS,我们提出幻觉不敏感模型编辑(HIME),一种简单有效的分层自适应权重编辑方法,可选择性修改潜在特征以抑制幻觉,同时保留预训练知识。大量实验表明,HIME在开放式生成基准(包括CHAIR、MME及GPT-4V辅助评估)上,平均降低61.8%的幻觉,且不引入额外参数、推理延迟或计算开销。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have demonstrated impressive multimodal understanding capabilities, yet they remain prone to object hallucination, where models describe non-existent objects or attribute incorrect factual information, raising serious concerns for reliable real-world deployment. While fine-tuning is a commonly adopted mitigation strategy, its high computational cost and practical difficulty motivate the need for training-free alternatives, among which model editing has recently emerged as a promising direction. However, indiscriminate editing risks disrupting the rich implicit knowledge encoded in pre-trained LVLMs, leading to a fundamental question: how much intervention is necessary at each layer to suppress hallucinations while preserving pre-trained knowledge? To address this question, we present a systematic analysis of LVLM decoders built on three widely used large language model backbones-Qwen, LLaMA, and Vicuna-revealing clear layer-wise differences in susceptibility to object hallucination. Building on these insights, we introduce the Hallucination Insensitivity Score (HIS), a principled metric that quantifies each layer's sensitivity to hallucination and provides guidance for targeted intervention. Leveraging HIS, we propose Hallucination Insensitivity Model Editing (HIME), a simple yet effective layer-adaptive weight editing approach that selectively modifies latent features to suppress hallucinations while preserving pre-trained knowledge. Extensive experiments demonstrate that HIME reduces hallucinations by an average of 61.8% across open-ended generation benchmarks, including CHAIR, MME, and GPT-4V-aided evaluation, without introducing additional parameters, inference-time latency, or computational overhead.

幻觉抑制模型编辑视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。