用多个检测器组合，减少视觉语言模型的幻觉错误。
Ensembling Multiple Hallucination Detectors Trained on VLLM Internal Representations
- 基于VLM内部表示训练逻辑回归检测器
- 集成多个检测器使幻觉率显著下降
- 适合追求答案准确性的VQA系统优化
本文介绍我们团队y3h2在KDD Cup 2025 Meta CRAG-MM挑战赛中的第五名解决方案。该挑战赛基于一个聚焦图像事实性问题的视觉问答（VQA）数据集，包含第一人称图像。评估以大语言模型自动评分的准确率为基准，错误回答会扣分。因此，我们的策略重点在于减少视觉语言模型内部表示引发的幻觉。具体方法是利用隐藏状态及特定注意力头输出，训练基于逻辑回归的幻觉检测模型，并采用模型集成策略。尽管少量正确答案被误判，但整体幻觉大幅降低，最终跻身排行榜前列。
原文摘要 · Abstract (English)
This paper presents the 5th place solution by our team, y3h2, for the Meta CRAG-MM Challenge at KDD Cup 2025. The CRAG-MM benchmark is a visual question answering (VQA) dataset focused on factual questions about images, including egocentric images. The competition was contested based on VQA accuracy, as judged by an LLM-based automatic evaluator. Since incorrect answers result in negative scores, our strategy focused on reducing hallucinations from the internal representations of the VLM. Specifically, we trained logistic regression-based hallucination detection models using both the hidden_state and the outputs of specific attention heads. We then employed an ensemble of these models. As a result, while our method sacrificed some correct answers, it significantly reduced hallucinations and allowed us to place among the top entries on the final leaderboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。