用高斯点云重建真实场景,实现零样本机器人操作的仿真到现实迁移。
High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting
- 结合高斯点云与网格物体,生成视觉逼真且物理准确的仿真环境。
- 通过多模态大模型自动推断物体物理属性与运动结构,提升仿真真实性。
- 训练策略在多个真实任务上实现零样本跨域成功,适合强化学习研究者。
机器人学习的可扩展性严重受限于真实世界数据采集的成本与人力投入。尽管仿真数据提供了可扩展的替代方案,但其往往因视觉外观、物理属性和物体交互方面的显著差异而难以泛化至真实世界。为此,我们提出 RoboSimGS,一种新型的 Real2Sim2Real 框架,将多视角真实图像转换为可扩展、高保真且具备物理交互能力的仿真环境。该方法采用混合表示:3D 高斯点云(3DGS)捕捉环境的逼真视觉外观,而交互物体则使用网格基元确保精确物理模拟。关键的是,我们首次引入多模态大语言模型(MLLM)自动化创建物理合理且可动的资产。该模型分析视觉数据,推断物体的物理属性(如密度、刚度)及复杂运动结构(如铰链、滑轨)。实验表明,完全基于 RoboSimGS 生成的数据训练的策略,在多种真实世界的操作任务中实现了成功的零样本仿真到现实迁移。此外,RoboSimGS 数据显著提升了现有顶尖方法的性能与泛化能力。结果验证了 RoboSimGS 作为弥合仿真与现实差距的强大且可扩展解决方案的有效性。
原文摘要 · Abstract (English)
The scalability of robotic learning is fundamentally bottlenecked by the significant cost and labor of real-world data collection. While simulated data offers a scalable alternative, it often fails to generalize to the real world due to significant gaps in visual appearance, physical properties, and object interactions. To address this, we propose RoboSimGS, a novel Real2Sim2Real framework that converts multi-view real-world images into scalable, high-fidelity, and physically interactive simulation environments for robotic manipulation. Our approach reconstructs scenes using a hybrid representation: 3D Gaussian Splatting (3DGS) captures the photorealistic appearance of the environment, while mesh primitives for interactive objects ensure accurate physics simulation. Crucially, we pioneer the use of a Multi-modal Large Language Model (MLLM) to automate the creation of physically plausible, articulated assets. The MLLM analyzes visual data to infer not only physical properties (e.g., density, stiffness) but also complex kinematic structures (e.g., hinges, sliding rails) of objects. We demonstrate that policies trained entirely on data generated by RoboSimGS achieve successful zero-shot sim-to-real transfer across a diverse set of real-world manipulation tasks. Furthermore, data from RoboSimGS significantly enhances the performance and generalization capabilities of SOTA methods. Our results validate RoboSimGS as a powerful and scalable solution for bridging the sim-to-real gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。