arXiv:2604.09338cs.AIcs.CL2026-04被引 1

提出空间推理新评估框架,揭示模型在逐步解题中的真实短板

Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym

  • 构建可逐步决策的2D网格寻路环境Spatial-Gym,支持回溯测试
  • 最强模型解题率仅16.0%,远低于人类98.0%,且越难任务越无法提升推理
  • 视觉输入使解题率下降73%,长思维链仍保持3-5倍优势

空间推理对导航与机器人至关重要,但现有评估多为一次性生成,与人类逐步交互方式不符。本文提出Spatial-Gym,一个基于Gymnasium的2D网格路径规划环境,将空间约束推理独立出来,以序列决策任务形式测试,支持可选回溯。在500个实验中,评估8个模型在三种设置(一次性、逐步、逐步+回溯)下的表现,对比人类、随机策略及A*基线。最优模型GPT-OSS 120B解题率为16.0%,较人类基线低82分。逐步格式使弱模型提升最多5.4%,但强模型因全局规划受限反而下降5.6%。回溯提升任务完成度,仅对弱模型提高解题率;强模型极少回溯,未受益。关键发现:(1) 模型无法随难度提升推理投入;(2) 接收环境图像的视觉模型解题率下降73%;(3) 延长思维链在逐步设置下仍保持3-5倍准确率优势。Spatial-Gym可用于诊断模型缺陷,并为强化学习改进空间推理提供框架。

原文摘要 · Abstract (English)

Spatial reasoning is central to navigation and robotics, yet measuring model capabilities on these tasks remains difficult. Existing benchmarks evaluate models in a one-shot setting, requiring full solution generation in a single response, unlike humans, who work in interactive environments step-by-step. We introduce Spatial-Gym, a Gymnasium environment that isolates spatial constraint reasoning by testing pathfinding in 2D-grid puzzles as a sequential decision task with optional backtracking. We evaluate eight models in three settings (one-shot, step-by-step, step-by-step with backtracking) against human, random, and A* baselines on 500 episodes. The best model, GPT-OSS 120B, achieves a solve rate of 16.0%, 82 points below the human baseline (98.0%). Step-by-step format helps weaker models (up to +5.4%) by removing formatting errors, but hurts stronger models (up to 5.6%) by constraining global planning. Backtracking improves episode completion, but increases solve rate only for weaker models; stronger models rarely backtrack and do not benefit from it. Our experiments have three key findings: (1) models fail to scale reasoning effort with difficulty, (2) vision models receiving images of the spatial environment reduce solve rate by 73%, and (3) extended chain-of-thought reasoning retains a 3-5x accuracy advantage over standard inference even in the step-by-step setting. Spatial-Gym enables diagnosis of model limitations and provides a framework for improving spatial reasoning through reinforcement learning.

空间推理评估基准强化学习链式思考

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。