arXiv:2608.09344cs.CV2026-08

不训练也不微调,按输入动态抑制视觉语言模型的幻觉。

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs

论文配图:Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs
图 1 · 摘自论文原文
  • 为不同幻觉类型构建独立子空间,实现细粒度修正。
  • 测试时自适应加权,对每张图选择最可能的幻觉方向抑制。
  • 无需训练,适配多模型、多数据集,高效稳定。

大型视觉语言模型(LVLM)虽具强大多模态推理能力,但常因幻觉导致文本描述与图像不符。现有方法中,模型编辑比解码调整更高效稳定,但通常依赖单一全局子空间,无法区分不同输入的多样幻觉模式。本文提出一种无需训练的幻觉缓解框架,在推理阶段动态实现每实例的幻觉抑制。首先构建多个解耦的幻觉子空间,分别对应不同幻觉类型;推理时根据输入与各子空间的相关性自适应计算权重,通过动态组合投影,精准抑制最可能的幻觉方向,同时保留图像相关语义。在多个视觉语言基准和模型家族上的实验表明,该方法显著提升性能,具备强鲁棒性、泛化性和效率。

原文摘要 · Abstract (English)

Recent advances in large vision-language models (LVLMs) have enabled powerful multimodal reasoning by integrating visual encoders with large language models (LLMs). However, their reliability is frequently undermined by hallucinations, where generated text inaccurately describes the visual input. Although fine-tuning can mitigate this problem, it is computationally expensive and requires large, curated datasets, making training-free alternatives attractive. Among these, model editing is more promising than decoding-based approaches: decoding methods adapt outputs per input but introduce computational overhead and instability, whereas model editing modifies internal representations offline, providing a more efficient and stable solution. However, existing model-editing techniques typically rely on a single global subspace to correct hallucinations, treating all test samples identically and failing to capture diverse hallucination modes across inputs. To address this limitation, we propose a training-free hallucination mitigation framework for dynamic, per-instance suppression at test time. Our method first constructs a set of Disentangled Hallucination Subspaces, each isolating a distinct hallucination mode. During inference, the model adaptively calculates weights reflecting each input's relationship to these subspaces, guiding a dynamically combined projection that selectively suppresses the most probable hallucination directions while preserving image-grounded semantics. Extensive experiments across multiple vision-language benchmarks and LVLM families demonstrate consistent improvements, highlighting the robustness, generalizability, and efficiency of our approach.

幻觉抑制模型编辑视觉语言无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。