拆解视觉语言模型答错题的原因，发现错误发生在识别前或识别后。
Failing to See or Failing to Know? Attributing Errors in Vision-Language Models
- 构建树状框架，区分错误是因识别失败还是知识缺失。
- 4个模型在2个数据集上显示：约30%错误在识别前发生，70%在识别后。
- 图像特征适合判断识别问题，提示状态可预测答案正确性。
视觉语言模型（VLMs）能在清晰图像中识别实体，却仍会在需要事实知识的问题上出错。以往研究要么孤立分析单一错误类型，要么将错误视为单一二元结果。本文提出一种树状结构框架，对知识密集型视觉问答中的失败进行模型特定的运作结果分类。在两个数据集和四个VLMs上，观察到一致的运作结果分布：部分错误发生在实体识别之前，部分则在识别相关实体后仍存在。视觉标记表示对识别决策最具信息量；提示隐藏状态能更有效预测答案成功，但事实访问归因仍困难，仅呈现微弱信号。这些生成前信号支持基于归因的路由，引导至针对性干预，包括图像修复、实体支持、问题重写和事实证据补充。
原文摘要 · Abstract (English)
Vision-language models (VLMs) can recognize entities in clear images yet still fail when answering questions that require factual knowledge beyond what is directly observable. Prior work has either examined individual failure modes in isolation or treated incorrect answers as monolithic, binary failures. We propose a tree-structured framework that organizes failures in knowledge-intensive visual question answering into model-specific operational outcomes. Across two datasets and four VLMs, we observe consistent distributions of operational outcomes: some failures occur before entity recognition, while others persist after the relevant entity is recognized. Visual token representations are most informative for recognition-related decisions. Prompt hidden states predict answer success more effectively, although factual-access attribution remains difficult and exhibits only a weak signal. These pre-generation signals support attribution-guided routing to targeted interventions, including image repair, entity support, question rewriting, and factual evidence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。