通过分层测试揭示大模型空间推理的短板,发现其常走捷径。
Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests

- 将三维堆叠物体计数拆解为9个感知与认知子任务,模拟人类发展过程。
- 8万张合成图像测试显示,顶级模型在整体任务上表现好但底层能力弱。
- 用思维链+可验证奖励训练,显著提升模型各层级一致性与准确率。
多模态大语言模型在视觉理解上表现优异,但在人类轻易完成的空间推理任务中表现不佳。现有基准将模型视为黑箱,难以判断性能低下是源于感知问题(如识别物体边界),还是认知问题(如通过遮挡推断隐藏结构)。我们提出Spatial-IQ,一个分层诊断框架,将堆叠3D结构中的物体计数分解为9个按人类空间认知发展阶段排列的感知与认知子任务,并以心理旋转作为额外探测目标。借助NVIDIA Isaac Sim,我们生成了约8万张包含每项任务真实标签的多样化3D结构图像数据集。我们在三种输出格式(自由文本、多选图像、图像编辑)下评估模型并对比人类基线。结果表明,顶尖模型虽能完成目标任务(物体计数),却往往未掌握支持该任务的底层子任务,且各模型对层级链条的保留程度各异,暴露出仅靠目标任务准确率无法察觉的捷径行为。最后,我们证明在分层子任务上使用思维链(CoT)监督,并结合可验证奖励的强化学习训练,能显著提升模型在子任务间的空间一致性及目标任务准确率,验证了该分解架构作为诊断工具与训练信号的双重价值。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) excel at visual interpretation but fail on spatial reasoning tasks that humans solve reliably. Existing benchmarks evaluate these models as black boxes, limiting their ability to identify the underlying causes of lower performance: when a model fails a spatial reasoning task, it remains difficult to ascertain whether the hurdle is perceptual, such as recognizing object boundaries, or cognitive, such as reasoning about occlusion to infer hidden geometry. We introduce Spatial-IQ, a hierarchical diagnostic framework that decomposes object counting in stacked 3D structures into 9 perceptual and cognitive sub-tasks organized by the developmental stages of human spatial cognition, with mental rotation as an additional target probe. Using NVIDIA Isaac Sim, we procedurally generated a diverse dataset of roughly 80,000 stacked 3D structures with per-task ground truth. We evaluate models across three output formats (free-response text, multiple-choice images, and image editing) alongside a human baseline. The Spatial-IQ framework shows that top-performing models often succeed at the target task (object counting) without succeeding on the lower-level sub-tasks intended to support it, and that models differ in how much of these hierarchical chains they preserve, often revealing shortcut behavior that raw target-task accuracy alone would obscure. Finally, we demonstrate that training models with chain-of-thought (CoT) supervision over our hierarchical sub-tasks, combined with reinforcement learning with verifiable rewards, significantly improves both spatial consistency across sub-tasks and target-task accuracy, supporting the value of the proposed decomposition as both a diagnostic tool and a training signal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。