用细粒度文本嵌入提升视觉模型幻觉检测能力
Do Vision Encoders Truly Explain Object Hallucination?: Mitigating Object Hallucination via Simple Fine-Grained CLIPScore
- 在名词级别引入文本嵌入，增强评估的细粒度
- 在OHD-Caps上比传统CLIPScore准确率高39.6%无额外训练
- 可用于数据过滤，降低大模型幻觉4.9%（POPE指标）
近年来，大型视觉语言模型（LVLMs）在多个领域表现出色，但普遍存在物体幻觉问题。本文在判别式检索型评测设置（OHD-Caps）下研究该问题，而非自由生成描述。我们重新审视了以往认为幻觉源于视觉编码器表征能力有限的观点，分析表明其并非检测幻觉的主要瓶颈。基于此，我们提出细粒度CLIPScore（F-CLIPScore），通过在名词级别融合文本嵌入，实现更精细的评估。在OHD-Caps基准上的实验显示，F-CLIPScore在无需额外训练的情况下，准确率较传统方法显著提升39.6%。进一步验证表明，基于F-CLIPScore的数据过滤可有效降低LVLM的幻觉程度，使对齐预训练后的POPE准确率提升4.9%。代码已开源。
原文摘要 · Abstract (English)
Recently, Large Vision-Language Models (LVLMs) show remarkable performance across various domains. However, these models suffer from object hallucination. In this work, we study object hallucination primarily in a discriminative, retrieval-style evaluation setting (OHD-Caps), rather than in free-form caption generation. This study revisits the previous claim that the cause of such hallucinations lies in the limited representational capacity of the vision encoder. Our analysis implies that the capacity of the vision encoder is not necessarily a major limiting factor in detecting object hallucination. Based on this insight, we propose Fine-grained CLIPScore (F-CLIPScore), a simple yet effective evaluation metric that enhances object-level granularity by incorporating text embeddings at the noun level. Evaluations on the OHD-Caps benchmark show that F-CLIPScore significantly outperforms conventional CLIPScore in accuracy by a large margin of 39.6% without additional training. We further demonstrate that F-CLIPScore-based data filtering reduces object hallucination in LVLM (4.9% in POPE accuracy after alignment pretraining). Our code is publicly available at https://github.com/abzb1/f-clip
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。