arXiv:2608.29924cs.CVcs.AI2026-08

通过隐式稳定特征,显著降低大模型幻觉问题。

Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization

论文配图:Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization
图 1 · 摘自论文原文
  • 在微调阶段构建对扰动的不变性,无需推理时额外计算
  • 在多个基准上使幻觉率降低46%-63%,性能全面提升
  • 适合追求低延迟、高准确的视觉语言应用开发者

大型视觉语言模型(LVLM)容易产生幻觉:会流畅描述图像中并不存在的对象、属性和场景。我们发现部分失败源于其表征的一个可度量特性——特征不稳定性:输入的轻微语义保持扰动会导致学习嵌入的大幅变化,幻觉率与这种变异性正相关。现有缓解方法为显式干预,需在推理时进行潜在空间引导或约束解码,带来每次查询的开销。本文提出隐式稳定策略:在微调阶段将扰动不变性嵌入模型权重,部署时无需额外操作。框架INFUSE首先以扰动平均值和真实标签作为锚点,稳定视觉与文本表征;再通过双向对比目标对齐跨模态表征。理论证明锚点均方根偏差随视图数K以1/√K速率缩小;在利普希茨解码器下,该性质限制了任意扰动对模型幻觉行为的影响。在LLaVA-1.5、LLaVA-1.6和Qwen3-VL-8B-Instruct上,INFUSE相较基线模型使AMBER CHAIR降低46%-63%,同时提升ObjHal、MMHal、HallusionBench和POPE指标,且保持VQA-v2和TextVQA性能,无推理开销。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) are prone to hallucinations: they fluently describe objects, attributes, and scenes that are not in the image. We connect part of this failure to a measurable property of their representations, feature instability, where mild semantics-preserving perturbations of the input cause large changes in the learned embeddings; hallucination rates rise together with this variability. Existing stability-motivated remedies are explicit, in the sense that they intervene at inference time through latent steering or constrained decoding, and pay for it on every query. We propose implicit stabilization instead: perturbation-invariance is built into the model weights during fine-tuning, and nothing extra runs at deployment. Our framework, INFUSE, first stabilizes visual and textual representations around perturbation-averaged and ground-truth anchors, then aligns the stabilized representations across modalities with bidirectional contrastive objectives. We prove that the anchor's root-mean-square deviation from the perturbation-mean representation shrinks at rate $1/\sqrt{K}$ in the number of views, and that under a Lipschitz decoder, this bounds how much any perturbation can change the model's hallucination behavior. On LLaVA-1.5, LLaVA-1.6, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR by 46-63% relative to each base model, improves ObjHal, MMHal, HallusionBench, and POPE, and preserves VQA-v2 and TextVQA, all with no inference-time overhead.

视觉语言幻觉抑制特征稳定微调优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。