让机器人用自然语言推理来指导操作,提升复杂任务的自主完成能力。
$R^3$: Training Robots to Reason in Natural Language via Reinforcement Learning

- 用自由语言推理替代结构化指令,通过强化学习训练视觉语言模型
- 在双臂购物打包任务中显著优于纯指令模仿学习基线
- 适合需要长期规划和纠错的机器人操作场景
语言推理使基础模型能在测试阶段投入更多计算资源处理复杂问题,如任务分解、约束追踪和未来后果预测。然而,这种机制是否能提升机器人操作性能尚不明确——长周期任务需追踪进展、理解物体关系、纠正错误并控制有噪声的底层策略。本文研究视觉语言模型(VLM)能否直接通过自然语言进行推理以指导低层操作策略。提出 $R^3$:一种简单的后训练方法,先在专家生成的推理轨迹上微调 VLM 以建立推理风格,再利用离线动作数据进行单步评分驱动的强化学习优化。与以往依赖结构化轨迹作为辅助监督的方法不同,$R^3$ 训练自由形式的语言推理,用于生成测试时的动作引导。我们在 Language Table 和模拟双臂购物打包两个受控实验平台验证了该方法。$R^3$ 显著提升了探索效率和跨未见任务的泛化能力,在两项基准上均大幅超越仅依赖指令的模仿学习基线。分析表明,自由语言推理可作为测试时计算资源调度机制,有效引导底层策略。项目页面:https://robotic-reasoner.github.io/
原文摘要 · Abstract (English)
Reasoning in language allows foundation models to spend more test-time compute on hard problems, such as those requiring decomposition, constraint tracking, and prediction of future consequences. Whether this mechanism can improve robotic manipulation remains unclear, where long-horizon tasks require tracking partial progress, reasoning about object relations, recovering from mistakes, and steering noisy low-level policies. In this paper, we study whether VLMs can be trained to reason directly in natural language to guide low-level manipulation policies. We introduce $R^3$, a simple post-training recipe that turns off-the-shelf VLMs into robotic reasoners: it first mid-trains a VLM on expert-generated reasoning traces to initialize the desired reasoning style, then improves the reasoner with single-step rubric-based RL from offline action data. Unlike prior robotic reasoning methods that mostly use structured traces as auxiliary supervision, $R^3$ trains free-form language reasoning to produce test-time guidance for action. We instantiate $R^3$ on Language Table and simulated bimanual grocery packing, two controlled testbeds for studying robotic reasoning and long-horizon manipulation. $R^3$ improves exploration and generalization across unseen tasks and significantly outperforms instruction-only imitation learning baselines on both benchmarks. Our analyses suggest that free-form language reasoning can function as a test-time compute mechanism for steering low-level policies. Our project page is available at https://robotic-reasoner.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。