用视频生成可编辑的机器人操作仿真,降低真实数据采集成本
RoboPearls: Editable Video Simulation for Robot Manipulation
- 基于3D高斯点云重建视频,生成逼真一致的仿真环境
- 支持多种物体操作,仿真效果在多个数据集上表现优异
- 结合大模型实现自然语言控制与学习问题自动分析
通用机器人操作策略的发展得益于大规模跨环境示范数据,但真实世界示范数据采集成本高、效率低,制约了数据获取的可扩展性。现有仿真平台虽能提供可控环境,但仿真到现实的差距仍难克服。为此,我们提出RoboPearls——一种用于机器人操作的可编辑视频仿真框架。基于3D高斯点云(3DGS),RoboPearls可从示范视频构建照片级真实的视图一致仿真,并支持多种仿真操作,包括各类物体操作,其核心由增量语义蒸馏(ISD)和3D正则化NNFM损失(3D-NNFM)模块驱动。通过引入大语言模型(LLM),RoboPearls实现用户友好的自然语言指令解析与自动化仿真生成。同时,利用视觉-语言模型(VLM)分析机器人学习问题,闭环优化仿真性能。我们在RLBench、COLOSSEUM、Ego4D、Open X-Embodiment等多个数据集及真实机器人上进行广泛实验,验证了RoboPearls的仿真有效性。
原文摘要 · Abstract (English)
The development of generalist robot manipulation policies has seen significant progress, driven by large-scale demonstration data across diverse environments. However, the high cost and inefficiency of collecting real-world demonstrations hinder the scalability of data acquisition. While existing simulation platforms enable controlled environments for robotic learning, the challenge of bridging the sim-to-real gap remains. To address these challenges, we propose RoboPearls, an editable video simulation framework for robotic manipulation. Built on 3D Gaussian Splatting (3DGS), RoboPearls enables the construction of photo-realistic, view-consistent simulations from demonstration videos, and supports a wide range of simulation operators, including various object manipulations, powered by advanced modules like Incremental Semantic Distillation (ISD) and 3D regularized NNFM Loss (3D-NNFM). Moreover, by incorporating large language models (LLMs), RoboPearls automates the simulation production process in a user-friendly manner through flexible command interpretation and execution. Furthermore, RoboPearls employs a vision-language model (VLM) to analyze robotic learning issues to close the simulation loop for performance enhancement. To demonstrate the effectiveness of RoboPearls, we conduct extensive experiments on multiple datasets and scenes, including RLBench, COLOSSEUM, Ego4D, Open X-Embodiment, and a real-world robot, which demonstrate our satisfactory simulation performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。