arXiv:2601.01957cs.CV2026-01被引 2

通过自适应事实引导编辑,减少大模型的视觉幻觉问题。

AFTER: Mitigating the Object Hallucination of LVLM via Adaptive Factual-Guided Activation Editing

  • 基于事实语义自适应调整模型内部激活,精准建模图文关联。
  • 在AMBER基准上最多降低16.3%的幻觉率,效果显著。
  • 适合关注多模态模型可信性与幻觉抑制的研究者使用。

大型视觉语言模型(LVLMs)在跨模态任务中取得了显著进展,但因语言偏见容易产生物体幻觉,主要分为类别、属性和关系幻觉,严重阻碍可信AI应用。现有激活编辑方法虽有效且成本低,但忽视了文本语义的事实引导作用,难以明确缓解语言偏见。为此,我们提出自适应事实引导的视觉-文本编辑方法AFTER,包含事实增强激活引导(FAS)与查询自适应偏移优化(QAO),以自适应地将原始有偏激活导向真实语义。FAS提供通用的事实引导,显式建模精确的视觉-文本关联;QAO引入查询感知偏移估计器,从通用引导向量生成特定查询的编辑,提升编辑的多样性和粒度。在三个主流LVLM上的标准幻觉基准测试表明,AFTER效果显著,尤其在AMBER基准上相比基线最高降低16.3%的幻觉率。代码与数据将公开以保证可复现性。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have achieved substantial progress in cross-modal tasks. However, due to language bias, LVLMs are susceptible to object hallucination, which can be primarily divided into category, attribute, and relation hallucination, significantly impeding the trustworthy AI applications. Editing the internal activations of LVLMs has shown promising effectiveness in mitigating hallucinations with minimal cost. However, previous editing approaches neglect the effective guidance offered by factual textual semantics, thereby struggling to explicitly mitigate language bias. To address these issues, we propose Adaptive Factual-guided Visual-Textual Editing for hallucination mitigation (AFTER), which comprises Factual-Augmented Activation Steering (FAS) and Query-Adaptive Offset Optimization (QAO), to adaptively guides the original biased activations towards factual semantics. Specifically, FAS is proposed to provide factual and general guidance for activation editing, thereby explicitly modeling the precise visual-textual associations. Subsequently, QAO introduces a query-aware offset estimator to establish query-specific editing from the general steering vector, enhancing the diversity and granularity of editing. Extensive experiments on standard hallucination benchmarks across three widely adopted LVLMs validate the efficacy of the proposed AFTER, notably achieving up to a 16.3% reduction of hallucination over baseline on the AMBER benchmark. Our code and data will be released for reproducibility.

幻觉抑制视觉语言模型激活编辑多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。