用大模型生成双臂灵巧操作数据,解决人形机器人训练数据少的问题。
HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning
- 通过原子操作和大模型推理生成空间约束,自动规划双臂动作
- 在新构建的基准上验证,扩散模型性能随数据量提升而增长
- 适合研究人形机器人灵巧操作与自动化数据生成的学者
在机器人操作领域,现有数据集和仿真基准主要面向机械臂平台。然而,配备双臂和灵巧手的人形机器人,其仿真任务与高质量示范数据严重不足。双臂灵巧操作本身更复杂,需协调手臂运动与手部操作,导致自主数据采集困难。本文提出HumanoidGen,一个基于大模型推理的自动化任务生成与示范收集框架,利用原子灵巧操作和大模型推理生成关系约束。具体而言,基于原子操作对物体与灵巧手进行空间标注,并通过大模型规划器根据物体属性与场景生成可执行的空间约束链。为进一步增强长程任务与标注不足情况下的规划能力,采用变体蒙特卡洛树搜索优化大模型推理。实验中,我们构建了包含增强场景的新基准以评估生成数据质量。结果表明,2D与3D扩散策略的性能可随生成数据集规模提升而扩展。项目主页:https://openhumanoidgen.github.io。
原文摘要 · Abstract (English)
For robotic manipulation, existing robotics datasets and simulation benchmarks predominantly cater to robot-arm platforms. However, for humanoid robots equipped with dual arms and dexterous hands, simulation tasks and high-quality demonstrations are notably lacking. Bimanual dexterous manipulation is inherently more complex, as it requires coordinated arm movements and hand operations, making autonomous data collection challenging. This paper presents HumanoidGen, an automated task creation and demonstration collection framework that leverages atomic dexterous operations and LLM reasoning to generate relational constraints. Specifically, we provide spatial annotations for both assets and dexterous hands based on the atomic operations, and perform an LLM planner to generate a chain of actionable spatial constraints for arm movements based on object affordances and scenes. To further improve planning ability, we employ a variant of Monte Carlo tree search to enhance LLM reasoning for long-horizon tasks and insufficient annotation. In experiments, we create a novel benchmark with augmented scenarios to evaluate the quality of the collected data. The results show that the performance of the 2D and 3D diffusion policies can scale with the generated dataset. Project page is https://openhumanoidgen.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。