arXiv:2512.24532cs.AIcs.CL2025-12被引 1

让大模型分步完成空间推理,通过分解动作提升规划能力

From Building Blocks to Planning: Multi-Step Spatial Reasoning in LLMs with Reinforcement Learning

  • 将空间推理拆解为旋转、平移等基础动作,分阶段训练
  • 在拼图环境中实现多步规划,动态与静态场景下均优于基线
  • 训练更快更稳定,适合需要复杂空间决策的应用

大语言模型在导航与规划任务中的空间推理能力日益受到关注。尽管具备强大的语言理解能力,但其在结构化环境中的空间变换与多步规划方面仍表现不足。本文提出一种两阶段方法,将空间推理分解为原子级基础动作及其组合。首先,对旋转、平移、缩放等基本空间变换进行监督微调,使模型具备基础物理感知能力;随后冻结该模型,在GRPO框架下使用轻量级LoRA适配器,以闭环方式学习如何组合这些基础动作完成多步规划任务。为此,我们构建了一个基于ASCII艺术的合成数据集及对应的强化学习环境。实验表明,该方法在动态(状态显式更新)和静态(依赖内部状态)两种环境下均显著优于通用骨干模型、物理感知模型及端到端强化学习模型。此外,该方法收敛更快,训练更稳定。最后,通过注意力模式分析验证了微调带来的空间理解提升。

原文摘要 · Abstract (English)

Spatial reasoning in large language models (LLMs) has gained increasing attention due to applications in navigation and planning. Despite strong general language capabilities, LLMs still struggle with spatial transformations and multi-step planning in structured environments. We propose a two-stage approach that decomposes spatial reasoning into atomic building blocks and their composition. First, we apply supervised fine-tuning on elementary spatial transformations, such as rotation, translation, and scaling, to equip the model with basic spatial physics. We then freeze this physics-aware model and train lightweight LoRA adapters within the GRPO framework to learn policies that compose these building blocks for multi-step planning in puzzle-based environments, in a closed-loop manner. To support this pipeline, we synthesize an ASCII-art dataset and construct a corresponding ASCII-based reinforcement learning environment. Our method consistently outperforms baselines, including the generic backbone, physics-aware model, and end-to-end RL models, under both Dynamic environments with explicit state updates and Static environments where the model must rely on its internal state across steps. In addition, the proposed approach converges faster and exhibits more stable training compared to end-to-end reinforcement learning from scratch. Finally, we analyze attention patterns to assess whether fine-tuning induces meaningful improvements in spatial understanding.

空间推理强化学习大模型多步规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。