通过指令嵌入检测多模态大模型的物体幻觉，无需额外训练。
Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Models
- 利用指令词嵌入隐含的视觉信息过滤错误视觉特征。
- 结合局部校准得分与上下文一致性得分，精准识别幻觉。
- 可直接插入现有模型，适合关注幻觉问题的研究者。
多模态大语言模型（MLLMs）虽取得显著进展，但物体幻觉仍是可靠部署的关键挑战。本文深入分析指令词嵌入，发现其隐含视觉信息，并能有效过滤误导性视觉嵌入带来的错误信息。基于此，提出指令透镜评分（InsLen），结合校准局部得分与对象词上下文一致性得分，作为即插即用的物体幻觉检测器，无需辅助模型或额外训练。在多个基准和多样化的MLLM架构上进行的广泛实验表明，InsLen始终优于现有幻觉检测方法，凸显其有效性与鲁棒性。代码已公开于https://github.com/Fraserlairh/Instruction-Lens-Score。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have achieved remarkable progress, yet the object hallucination remains a critical challenge for reliable deployment. In this paper, we present an in-depth analysis of instruction token embeddings and reveal that they implicitly encode visual information while effectively filtering erroneous information introduced by misleading visual embeddings. Building on this insight, we propose the Instruction Lens Score (InsLen), which combines a Calibrated Local Score with a Context Consistency Score that measures context consistency of the object tokens. The proposed approach serves as a plug-and-play object hallucination detector without relying on auxiliary models or additional training. Extensive experiments across multiple benchmarks and diverse MLLM architectures demonstrate that InsLen consistently outperforms existing hallucination detection methods, highlighting its effectiveness and robustness. The code is available at https://github.com/Fraserlairh/Instruction-Lens-Score.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。