提出几何解耦方法,用对抗训练减少视觉语言模型幻觉。
Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigation

- 通过极小极大博弈学习幻觉相关方向,分离潜在空间
- 在四个基准上平均提升POPE准确率6%以上,AMBER提升6%
- 无需训练即可部署,适合追求可靠性与通用性的研究者
大型视觉语言模型(LVLM)虽提升了多模态理解能力,但其可靠性受限于幻觉问题——生成内容与视觉事实冲突。现有缓解方法或依赖昂贵的外部干预(如指令微调、检索),或使用内部机制,却受制于有缺陷的注意力权重和纠缠的隐藏表示。本文提出对抗正交解耦(AOD),一种用于缓解LVLM幻觉的潜在空间几何框架。AOD通过极小极大目标学习幻觉相关方向:分类器将幻觉信号集中投影到特定分量,而对抗模块通过梯度反转层从正交残差空间中移除这些信号。由此学习的方向支持无需训练的双前向传播对比解码策略,在抑制幻觉的同时保持通用能力。在三个LVLM上对四个幻觉与四个效用基准的实验表明,AOD持续优于强基线,平均提升POPE准确率超6%,AMBER提升6%,并在如MMMU等效用任务上保持优异表现。进一步分析显示其跨数据集鲁棒迁移,说明AOD捕捉的是通用幻觉偏差而非特定数据集伪影。代码与数据集已开源。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have advanced multimodal understanding, yet their reliability is limited by hallucination, where generated content conflicts with visual facts. Existing mitigation methods either rely on costly external interventions, such as instruction tuning and retrieval, or use internal mechanisms that remain limited by flawed attention weights and entangled hidden representations. We propose Adversarial Orthogonal Disentanglement (AOD), a latent geometric framework for mitigating LVLM hallucinations. AOD learns a hallucination-related direction through a minimax objective: a classifier concentrates hallucination signals into the projected component, while an adversary removes them from the orthogonal residual space via a Gradient Reversal Layer. The learned direction enables a training-free dual-forward-pass contrastive decoding strategy that suppresses hallucinations while preserving general capabilities. Experiments on three LVLMs across four hallucination and four utility benchmarks show that AOD consistently outperforms strong baselines. It improves POPE accuracy by over 6\% on average, boosts AMBER by 6\%, and maintains strong performance on utility tasks such as MMMU. Further analysis shows robust transfer across datasets, suggesting that AOD captures general hallucination-related biases rather than dataset-specific artifacts. Our source code and datasets are available at https://github.com/Hunter-Wrynn/AOD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。