提出新方法在不降性能前提下减少视觉语言模型幻觉
Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation

- 通过语义感知解耦提取纯幻觉成分
- 仅更新与幻觉相关参数,减少误伤
- 在多个数据集上显著降幻觉,保留97.4%生成能力
大型视觉语言模型虽具强大生成能力,但常产生幻觉影响输出可靠性。基于标注数据的微调虽有效但计算成本高,近期代表征方法虽高效,却因幻觉成分提取不全及参数更新无差别导致通用生成能力下降。为此,本文提出双阶段框架MPD,核心为:(1)语义感知组件解耦以提取纯净幻觉成分;(2)可解释的参数更新机制,仅修改与幻觉最相关的参数。大量实验表明,MPD在LLaVA-Bench和MME上将幻觉降低23.4%,同时保持97.4%的通用生成能力,且无需额外计算开销。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) exhibit powerful generative capabilities but frequently produce hallucinations that compromise output reliability. Fine-tuning on annotated data devoid of hallucinations offers the most direct solution, while its high computational cost motivates recent representation-based methods, which focus on mitigating hallucinatory components within hidden representations. Though efficient, we empirically observe that these methods degrade general generation capacity due to incomplete extraction of hallucination components and non-selective parameter updates. To address these limitations, we propose MPD, a dual-stage framework for mitigating hallucinations without performance degradation. Specifically, our MPD relies on two essential factors: (1) semantic-aware component disentanglement to extract pure hallucination components, and (2) interpretable parameter updates that selectively modify parameters most relevant to hallucination. Extensive experiments demonstrate that MPD achieves state-of-the-art performance, reducing hallucinations by 23.4\% while maintaining 97.4\% of general generative capability as evaluated on LLaVA-Bench and MME, with no additional computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。