剖析视觉语言模型在组合式问答中的失败原因,发现四类核心错误模式。
How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA

- 按推理操作分解模型错误,定位失败源头与传播路径。
- 发现物体选择错在前馈计算,空间关系错在深层注意力。
- 适合研究多媒体推理故障诊断的学者和工程师。
组合式视觉问答要求视觉-语言模型(VLM)执行物体选择、空间关系解析和属性验证等多重推理操作。尽管整体表现良好,但其失败机制仍不明确。本文提出一种以操作为中心的机械分析框架,从推理操作起源和内部计算路径两方面分解失败。分析揭示四种主导失败模式:定位失败、推理失败、属性提取失败和语言先验主导,每种均对应视觉定位强度与答案正确性间的特定关系。通过在所有Transformer层实施三种互补因果干预,发现物体选择失败主要关联前馈计算,多步关系失败源于深层直接注意力,属性提取失败则与答案位置前馈计算相关。在VSR上的验证表明,单步空间错误集中于物体位置编码,区别于多步关系组合。这些发现揭示了不同操作类型的计算瓶颈,为多媒体推理中模型失败的精准诊断提供了理论基础。
原文摘要 · Abstract (English)
Compositional visual question answering requires Vision-Language Models (VLMs) to execute multiple reasoning operations like object selection, spatial relation resolution, and attribute verification. Despite strong aggregate performance, the mechanistic basis of VLM failures on this task remains underexplored. To address this gap, we analyze vision-operation misalignment in VLMs by examining how failures relate to specific reasoning operations and the internal computational pathways through which they arise and propagate. We introduce an Operation-centric mechanistic framework that decomposes VLM failures by both the reasoning operation where they originate and the internal computational pathway through which they propagate. Our analysis reveals four dominant failure modes: grounding failure, reasoning failure, attribute extraction failure, and language-prior dominance, each characterized by a distinct relationship between visual grounding strength and answer correctness. Through three complementary causal interventions applied across all transformer layers, we find that object-selection failures are associated primarily with feedforward computation, multi-step relational failures with late-layer direct attention, and attribute-extraction failures with answer-position feedforward computation. Validation on VSR further shows that single-step spatial failures are concentrated at object-position encoding, distinguishing them from multi-step relational composition. These findings reveal distinct computational bottlenecks across operation types and provide a principled basis for targeted diagnosis of VLM failures in multimedia reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。