arXiv:2608.05616cs.CV2026-08中稿 · ECCV

通过自评估分数检测视觉语言模型中的对象幻觉,无需额外模型或计算开销。

TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs

论文配图:TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs
图 1 · 摘自论文原文
  • 用特殊标记作为参照,从语言头提取每个对象的真值得分。
  • 在多个模型上实现超越现有方法17%以上的检测性能(如Qwen2.5-VL-7B)。
  • 仅需少量类别训练即可泛化到更大标签空间,适合实际部署场景。

尽管大型视觉语言模型(LVLMs)取得了显著进展,对象幻觉仍是阻碍其可信部署的根本性挑战。关键发现表明:真实与幻觉对象标记在隐藏表示中具有明显可分性,但在语言建模头处这种可分性基本丧失。我们提出TruthLens,一种自评估框架,使语言模型头在无需辅助模型或额外推理成本的情况下,暴露每个对象的真值信号。具体而言,将一个较少使用的特殊标记重用于参考。对于每个对象标记位置,提取语言模型头对该特殊标记分配的对数概率,并以其与预设常数的差值定义为真值分数。模型通过最小均方误差目标进行微调,使真实对象得分趋近于1,幻觉对象得分趋近于0,同时通过发散约束保留原始生成能力。尽管仅在有限对象类别上训练,TruthLens在包含更大标签空间的基准测试中仍表现出良好泛化性。在多个LVLM上的广泛实验表明其达到当前最优性能;尤其在Qwen2.5-VL-7B上,其在MS-COCO上的AUROC指标优于此前最佳方法超过17%。代码已开源:https://github.com/wyqstan/TruthLens。

原文摘要 · Abstract (English)

Despite the remarkable progress of large vision language models (LVLMs), object hallucination remains a fundamental challenge that hinders their trustworthy deployment. A key finding motivates our work: real and hallucinated object tokens are clearly separable in hidden representations, yet this separability is largely lost at the language-modeling (LM) head. We propose TruthLens, a self-evaluation framework that teaches the LM head to expose a per-object truthfulness signal without any auxiliary model or additional inference cost. Concretely, a rarely-used special token is repurposed as a reference token. For each object-token position, we extract the log-probability assigned to this special token by the LM head, and define its difference from a predefined constant as the truthfulness score. The model is then fine-tuned with an MSE objective that drives scores toward 1 for real objects and 0 for hallucinated ones, while a divergence constraint preserves the original generation capability. Despite being trained on only a limited set of object categories, TruthLens generalizes effectively to benchmarks with substantially larger label spaces. Extensive experiments across multiple LVLMs demonstrate state-of-the-art performance; notably, on Qwen2.5-VL-7B, TruthLens outperforms the previous best method on MS-COCO by over 17\% in AUROC. Our code is available at https://github.com/wyqstan/TruthLens.

幻觉检测视觉语言模型自评估真值评分

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。