诊断多模态模型在抽象视觉推理中‘知规则却选错答案’的问题。
StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning

- 用同一视觉线索设计三问任务,定位感知、归纳、匹配各环节错误
- 24个模型中22个存在‘规则正确但答案错误’现象,51.2%错在规则到实例的映射
- 发现规则与实例绑定是瓶颈,扩大模型或思考模式均无效
多模态大语言模型常能描述图像并识别规律,却仍无法选出正确答案。现有评测将感知、规则归纳和答案选择合并为单一对错信号,无法定位失败环节。本文提出StemBind,一个共享视觉茎干的诊断基准,通过三个对齐问题(感知、规则、完整)在同一证据上测试:模型可正确感知(P)和归纳规则(R),但完成任务(F)仍错51.2%。该基准涵盖9种可审计视觉操作,共2,298个轻知识题干,总计19,533个P/R/F任务,并基于Sternberg四阶段推理模型标注每个步骤。评估24个前沿MLLM配置发现:(i) 22/24模型存在“规则准确但最终错误”现象;(ii) 即使P与R正确,仍51.2%在匹配阶段出错;(iii) 主要瓶颈在规则到实例的映射(S3阶段);(iv) 模型规模扩大或启用思维链反而降低规则与完整任务准确率。研究揭示抽象视觉推理的断点在于规则-实例绑定,呼吁聚焦此环节改进。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) often know the rule but pick the wrong answer: on abstract visual reasoning (AVR) tasks, a model can describe what it sees and name the underlying pattern, yet still fail to choose the matching candidate. Existing AVR benchmarks cannot detect this because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal. We introduce StemBind, a shared-stem diagnostic benchmark that probes the same visual stem with three aligned questions: Perception (what is in the image), Rule (what pattern governs it), and Full (which option completes it), so a final-answer error can be attributed to a specific sub-step on the same evidence. StemBind contains 2,298 curated knowledge-light stems across nine auditable visual operations, totaling 19,533 P/R/F tasks, with each full item annotated by Sternberg's four reasoning stages (S1 Encode, S2 Infer, S3 Map, S4 Apply). Evaluating 24 frontier MLLM configurations yields four findings. (i) The R-F chasm: rule accuracy exceeds full-item accuracy on 22 of 24 models, so most failures happen after the rule is identified. (ii) A persistent binding gap: even when P and R are both correct on the same stem, models still answer F incorrectly 51.2% of the time. (iii) The bottleneck is S3: process diagnostics and Stage-wise Stimulus Augmentation localize the dominant failure to rule-to-instance mapping. (iv) Scaling and thinking do not help: neither larger models nor explicit thinking mode reliably closes the gap, and thinking even lowers rule and full-item accuracy. StemBind reframes AVR evaluation from final-answer ranking to locating where abstract visual reasoning breaks down, identifying rule-to-instance binding as a concrete next target for vision-grounded reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。