让机器人像人一样在空间中推理行动，提升适应与可解释性。
MolmoAct: Action Reasoning Models that can Reason in Space
- 分三阶段处理：感知→空间规划→精准动作，实现可编辑轨迹
- 真实世界任务成功率超基线10%-22.7%，零样本准确率达70.5%
- 开源1万条高质量轨迹数据集，支持开放研究与模型迭代
推理是目的性行为的核心，但多数机器人基础模型将感知与指令直接映射为控制，限制了适应性、泛化能力与语义理解。我们提出动作推理模型（ARMs），通过结构化三阶段流程整合感知、规划与控制。MolmoAct 将观测与指令编码为带深度感知的标记，生成可编辑的空间轨迹规划，并预测精确低层动作，实现可解释且可调控的行为。MolmoAct-7B-D 在仿真与真实场景中表现优异：在 SimplerEnv 视觉匹配任务上零样本准确率达 70.5%，超越闭源模型 Pi-0 与 GR00T N1.5；在 LIBERO 数据集上平均成功率达 86.6%，长程任务较 ThinkAct 提升 6.3%；真实世界微调中，单臂任务进度提升 10%，双臂任务提升 22.7%。其在分布外泛化上优于基线 23.3%，且在开放式指令遵循与轨迹调控中获最高人类偏好评分。我们首次发布 MolmoAct 数据集——包含超过 10,000 条多样化任务下的高质量机器人轨迹。使用该数据集训练使模型整体性能平均提升 5.5%。我们开源全部模型权重、训练代码、收集数据集与动作推理数据集，确立 MolmoAct 为先进机器人基础模型，并提供构建结构化动作推理模型的开放蓝图。
原文摘要 · Abstract (English)
Reasoning is central to purposeful action, yet most robotic foundation models map perception and instructions directly to control, which limits adaptability, generalization, and semantic grounding. We introduce Action Reasoning Models (ARMs), a class of robotic foundation models that integrate perception, planning, and control through a structured three-stage pipeline. Our model, MolmoAct, encodes observations and instructions into depth-aware perception tokens, generates mid-level spatial plans as editable trajectory traces, and predicts precise low-level actions, enabling explainable and steerable behavior. MolmoAct-7B-D achieves strong performance across simulation and real-world settings: 70.5% zero-shot accuracy on SimplerEnv Visual Matching tasks, surpassing closed-source Pi-0 and GR00T N1.5; 86.6% average success on LIBERO, including an additional 6.3% gain over ThinkAct on long-horizon tasks; and in real-world fine-tuning, an additional 10% (single-arm) and an additional 22.7% (bimanual) task progression over Pi-0-FAST. It also outperforms baselines by an additional 23.3% on out-of-distribution generalization and achieves top human-preference scores for open-ended instruction following and trajectory steering. Furthermore, we release, for the first time, the MolmoAct Dataset -- a mid-training robot dataset comprising over 10,000 high quality robot trajectories across diverse scenarios and tasks. Training with this dataset yields an average 5.5% improvement in general performance over the base model. We release all model weights, training code, our collected dataset, and our action reasoning dataset, establishing MolmoAct as both a state-of-the-art robotics foundation model and an open blueprint for building ARMs that transform perception into purposeful action through structured reasoning. Blogpost: https://allenai.org/blog/molmoact
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。