用3D高斯点云构建逼真机器人操作模拟环境,实现无需真实机器人的闭环训练与评估。
GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation
- 将高斯点云与物理引擎结合,生成逼真场景并支持机器人动作仿真
- 构建包含3种机器人和40多个物体的GSDF场景数据库,支持零样本跨域迁移
- 适合做视觉强化学习、远程操控数据采集及真实策略的可复现测试
本文提出GSWorld,一种用于机器人操作的鲁棒且逼真的模拟系统,融合3D高斯点云(Gaussian Splatting)与物理引擎。该框架倡导“闭环”开发:利用真实机器人数据训练策略,并在无须实际使用机器人的情况下完成可复现的评估与模拟到现实(sim2real)训练。为实现多样化场景的逼真渲染,我们提出新型资产格式GSDF(Gaussian Scene Description File),将高斯点云与网格表示结合,集成机器人URDF模型及其他物体。通过简化的重建流程,我们构建了一个包含3种机器人形态(单臂与双臂)以及超过40个物体的GSDF数据库。结合物理引擎,我们展示了五项直接应用:(1) 基于逼真图像的零样本模拟到现实像素到动作策略学习;(2) 自动化高质量DAgger数据收集以适应部署环境;(3) 在模拟中可复现地评测真实机器人策略;(4) 通过虚拟遥操作进行模拟数据采集;(5) 零样本模拟到现实视觉强化学习。项目官网:https://3dgsworld.github.io/。
原文摘要 · Abstract (English)
This paper presents GSWorld, a robust, photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines. Our framework advocates "closing the loop" of developing manipulation policies with reproducible evaluation of policies learned from real-robot data and sim2real policy training without using real robots. To enable photo-realistic rendering of diverse scenes, we propose a new asset format, which we term GSDF (Gaussian Scene Description File), that infuses Gaussian-on-Mesh representation with robot URDF and other objects. With a streamlined reconstruction pipeline, we curate a database of GSDF that contains 3 robot embodiments for single-arm and bimanual manipulation, as well as more than 40 objects. Combining GSDF with physics engines, we demonstrate several immediate interesting applications: (1) learning zero-shot sim2real pixel-to-action manipulation policy with photo-realistic rendering, (2) automated high-quality DAgger data collection for adapting policies to deployment environments, (3) reproducible benchmarking of real-robot manipulation policies in simulation, (4) simulation data collection by virtual teleoperation, and (5) zero-shot sim2real visual reinforcement learning. Website: https://3dgsworld.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。