用生成式模拟自动生成无限训练数据,让机器人学会目标驱动的抓取操作。
ForeRobo: Unlocking Infinite Simulation Data for 3D Goal-driven Robotic Manipulation
- 构建自洽的‘提方案-生成-学习-执行’循环,用生成环境替代直接学策略。
- 在多种刚体和可动物体任务中,相比顶尖模型提升56.32%的生成准确率。
- 真实世界测试中零样本迁移到20多个任务,平均成功率达79.28%,适合复杂场景部署。
高效利用仿真获取先进操作技能既具挑战性又意义重大。我们提出 ForeRobo,一种生成式机器人智能体,通过生成式仿真自主获取以目标状态为驱动的操作技能。不同于直接学习底层策略,该方法融合生成范式与经典控制。智能体首先提出待学习技能并构建对应仿真环境;随后配置物体以生成与技能一致的目标状态(ForeGen)。这些虚拟生成的无限数据用于训练所提出的状态生成模型(ForeFormer),该模型基于场景状态与任务指令,预测当前状态中每个点的3D目标位置,建立点对点对应关系。最后,采用经典控制算法在真实环境中执行动作。相较于端到端策略学习,ForeFormer具备更强可解释性与执行效率。我们在多种刚体与可动物体操作任务上训练并评估 ForeFormer,平均性能较当前最优模型提升56.32%,展现良好泛化能力。真实世界测试涵盖20余项任务,ForeRobo实现零样本仿生到现实迁移,平均成功率达79.28%。
原文摘要 · Abstract (English)
Efficiently leveraging simulation to acquire advanced manipulation skills is both challenging and highly significant. We introduce \textit{ForeRobo}, a generative robotic agent that utilizes generative simulations to autonomously acquire manipulation skills driven by envisioned goal states. Instead of directly learning low-level policies, we advocate integrating generative paradigms with classical control. Our approach equips a robotic agent with a self-guided \textit{propose-generate-learn-actuate} cycle. The agent first proposes the skills to be acquired and constructs the corresponding simulation environments; it then configures objects into appropriate arrangements to generate skill-consistent goal states (\textit{ForeGen}). Subsequently, the virtually infinite data produced by ForeGen are used to train the proposed state generation model (\textit{ForeFormer}), which establishes point-wise correspondences by predicting the 3D goal position of every point in the current state, based on the scene state and task instructions. Finally, classical control algorithms are employed to drive the robot in real-world environments to execute actions based on the envisioned goal states. Compared with end-to-end policy learning methods, ForeFormer offers superior interpretability and execution efficiency. We train and benchmark ForeFormer across a variety of rigid-body and articulated-object manipulation tasks, and observe an average improvement of 56.32\% over the state-of-the-art state generation models, demonstrating strong generality across different manipulation patterns. Moreover, in real-world evaluations involving more than 20 robotic tasks, ForeRobo achieves zero-shot sim-to-real transfer and exhibits remarkable generalization capabilities, attaining an average success rate of 79.28\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。