通过视觉对比编辑,零成本抑制大模型幻觉生成。
VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editing

- 用视觉对比扰动分析模型响应,定位幻觉区域
- 基于SVD分解激活模式,精准抑制幻觉子空间
- 无需微调或标签数据,适合医疗、自动驾驶等场景
大型视觉语言模型(LVLMs)常出现物体幻觉(OH),即生成图像中不存在的物体描述。这在医学影像和自动驾驶等关键应用中尤为严重。研究表明,幻觉源于语言先验:预训练中学习到的统计共现偏见。为此,我们提出视觉对比编辑(VCE),一种后处理方法,通过分析模型对对比视觉扰动的响应,识别并抑制幻觉倾向。利用奇异值分解(SVD)分解模型激活模式,分离出幻觉子空间,并针对性地修改参数以削弱其影响。与需微调或标注数据的方法不同,VCE为无标签干预,具备可扩展性和部署实用性。实验表明,VCE在多个基准上有效降低物体幻觉,同时保持原有计算效率。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) frequently suffer from Object Hallucination (OH), wherein they generate descriptions containing objects that are not actually present in the input image. This phenomenon is particularly problematic in real-world applications such as medical imaging and autonomous driving, where accuracy is critical. Recent studies suggest that the hallucination problem may stem from language priors: biases learned during pretraining that cause LVLMs to generate words based on their statistical co-occurrence. To mitigate this problem, we propose Visual Contrastive Editing (VCE), a novel post-hoc method that identifies and suppresses hallucinatory tendencies by analyzing the model's response to contrastive visual perturbations. Using Singular Value Decomposition (SVD), we decompose the model's activation patterns to isolate hallucination subspaces and apply targeted parameter edits to attenuate its influence. Unlike existing approaches that require fine-tuning or labeled data, VCE operates as a label-free intervention, making it both scalable and practical for deployment in resource-constrained settings. Experimental results demonstrate that VCE effectively reduces object hallucination across multiple benchmarks while maintaining the model's original computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。