arXiv:2411.11839cs.ROcs.CV2024-11被引 78

用3D高斯点云构建真实感机器人模拟器,实现高效数据生成与零样本迁移。

RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator

  • 基于3D高斯点云与物理引擎,构建真实到仿真再到真实的闭环模拟系统。
  • 在新视角和新场景下,模拟数据在真实机器人上表现优于真实数据模型。
  • 适合需要低成本大规模数据生成的机器人策略学习研究者使用。

真实世界具身数据的高效获取日益重要。然而,远程操作捕获的大规模示范成本极高,难以高效扩展数据量。在仿真环境中采样轨迹是大规模数据收集的可行方案,但现有模拟器在纹理与物理建模上难以实现高保真。为此,我们提出RoboGSim,一个基于3D高斯点云与物理引擎的真实到仿真再到真实(real2sim2real)的机器人模拟器。RoboGSim包含四个部分:高斯重建器、数字孪生构建器、场景组合器和交互引擎,可合成包含新视角、新物体、新轨迹与新场景的仿真数据。该系统支持在线、可复现且安全的操纵策略评估。真实到仿真与仿真到真实迁移实验显示纹理与物理高度一致。在RoboGSim与真实机器人平台上的测试结果表明,仅用模拟数据训练的模型可在真实机器人上实现零样本性能,效果接近真实数据。在新视角与新场景实验中,模拟数据模型的表现甚至优于真实数据模型。这不仅缩小了仿真到真实差距,也缓解了真实数据采集单源、高成本的局限。我们希望RoboGSim能成为政策学习公平比较的闭环模拟平台。

原文摘要 · Abstract (English)

Efficient acquisition of real-world embodied data has been increasingly critical. However, large-scale demonstrations captured by remote operation tend to take extremely high costs and fail to scale up the data size in an efficient manner. Sampling the episodes under a simulated environment is a promising way for large-scale collection while existing simulators fail to high-fidelity modeling on texture and physics. To address these limitations, we introduce the RoboGSim, a real2sim2real robotic simulator, powered by 3D Gaussian Splatting and the physics engine. RoboGSim mainly includes four parts: Gaussian Reconstructor, Digital Twins Builder, Scene Composer, and Interactive Engine. It can synthesize the simulated data with novel views, objects, trajectories, and scenes. RoboGSim also provides an online, reproducible, and safe evaluation for different manipulation policies. The real2sim and sim2real transfer experiments show a high consistency in the texture and physics. We compared the test results of RoboGSim data and real robot data on both RoboGSim and real robot platforms. The experimental results show that the RoboGSim data model can achieve zero-shot performance on the real robot, with results comparable to real robot data. Additionally, in experiments with novel perspectives and novel scenes, the RoboGSim data model performed even better on the real robot than the real robot data model. This not only helps reduce the sim2real gap but also addresses the limitations of real robot data collection, such as its single-source and high cost. We hope RoboGSim serves as a closed-loop simulator for fair comparison on policy learning. More information can be found on our project page https://robogsim.github.io/.

机器人模拟高斯点云零样本迁移数据生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。