用多模态大模型自动生成机器人仿真数据，提升训练效率与现实迁移能力。
GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
- 利用具备推理能力的多模态大模型自动构建复杂仿真任务和场景。
- 可生成100个关节物体任务、200个对象的演示数据，人力需求显著降低。
- 新提出的点云感知变压器模型支持零样本迁移，性能比纯真实数据训练高20%。
当前机器人仿真面临难以规模化的问题，主要因创建多样化任务和场景需大量人工参与。同时，仿真训练策略也受限于多数模拟到现实的方法仅针对单一任务。为此，本文提出GenSim2框架，利用具备编码、多模态和推理能力的大语言模型，自动生成包含长时序、带关节物体的复杂且真实的仿真任务。为大规模生成这些任务的示范数据，我们设计了可在物体类别内泛化的规划与强化学习求解器。该流程可生成最多100个关节类任务、200个不同物体的演示数据，大幅减少人工投入。为有效利用此类数据，我们提出一种多任务语言条件下的本体感知点云变换器（PPT）策略架构，能从生成的数据中学习，并展现出强大的模拟到现实零样本迁移能力。结合该流水线与策略架构，实验表明生成数据可用于零样本迁移或与真实数据联合训练，相较仅使用有限真实数据训练，策略性能提升20%。
原文摘要 · Abstract (English)
Robotic simulation today remains challenging to scale up due to the human efforts required to create diverse simulation tasks and scenes. Simulation-trained policies also face scalability issues as many sim-to-real methods focus on a single task. To address these challenges, this work proposes GenSim2, a scalable framework that leverages coding LLMs with multi-modal and reasoning capabilities for complex and realistic simulation task creation, including long-horizon tasks with articulated objects. To automatically generate demonstration data for these tasks at scale, we propose planning and RL solvers that generalize within object categories. The pipeline can generate data for up to 100 articulated tasks with 200 objects and reduce the required human efforts. To utilize such data, we propose an effective multi-task language-conditioned policy architecture, dubbed proprioceptive point-cloud transformer (PPT), that learns from the generated demonstrations and exhibits strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the policy architecture, we show a promising usage of GenSim2 that the generated data can be used for zero-shot transfer or co-train with real-world collected data, which enhances the policy performance by 20% compared with training exclusively on limited real data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。