不训练模型,通过删除关键文本词缓解视觉语言模型幻觉
CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models
- 用选择性删除文本词构建幻觉模型,捕捉幻觉源头
- 在六个基准上提升CHAI R分数20%,效果持续稳定
- 适合需要快速部署、无需训练的幻觉抑制场景
尽管大型视觉语言模型(LVLMs)发展迅速,但其生成幻觉内容的问题始终存在,影响实际应用的可靠性。现有无需训练的方法存在两大局限:(i) 对幻觉成因假设过窄;(ii) 在生成末期幻觉高发阶段效果下降。常见做法是通过完全或部分移除视觉标记来构建幻觉模型,并与原模型对比,但此法仍不足以阻止视觉信息向文本传播。基于此,我们提出一种新幻觉模型,通过选择性移除关键文本标记来捕获幻觉效应。进一步提出广义对比解码,整合多个幻觉模型以覆盖多样幻觉来源。二者结合形成CRoPS框架,在六项基准测试和三个LVLM家族中实现平均20%的CHAI R分数提升,优于当前最优无训练方法。
原文摘要 · Abstract (English)
Despite the rapid success of Large Vision-Language Models (LVLMs), a persistent challenge is their tendency to generate hallucinated content, undermining reliability in real-world use. Existing training-free methods address hallucinations but face two limitations: (i) they rely on narrow assumptions about hallucination sources, and (ii) their effectiveness declines toward the end of generation, where hallucinations are most likely to occur. A common strategy is to build hallucinated models by completely or partially removing visual tokens and contrasting them with the original model. Yet, this alone proves insufficient, since visual information still propagates into generated text. Building on this insight, we propose a novel hallucinated model that captures hallucination effects by selectively removing key text tokens. We further introduce Generalized Contrastive Decoding, which integrates multiple hallucinated models to represent diverse hallucination sources. Together, these ideas form CRoPS, a training-free hallucination mitigation framework that improves CHAIR scores by 20% and achieves consistent gains across six benchmarks and three LVLM families, outperforming state-of-the-art training-free methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。