用3D高斯点云生成新示范,让机器人只看一次就能应对多种变化。
Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulation
- 直接操作3D高斯点云生成多样真实示范
- 单次训练成功率达87.8%,远超传统方法的57.2%
- 适合需要少样本、强泛化的机器人操控任务
通过遥操作收集视觉运动策略数据面临耗时长、成本高和数据多样性不足的问题。现有方法在RGB空间进行图像增强或依赖物理模拟器的现实-仿真-现实流程,但前者受限于2D增强,后者因几何重建不准导致仿真不精确。本文提出RoboSplat,通过3D高斯点云(3DGS)重建场景,直接编辑重建结果,实现六类泛化下的数据增强:3D高斯替换(改变物体类型、场景外观、机器人形态)、等变变换(不同物体姿态)、视觉属性编辑(光照变化)、新视角合成(新相机位置)、3D内容生成(多样化物体)。真实世界实验表明,RoboSplat显著提升视觉运动策略的泛化能力。相比需数百次真实示范加2D增强的策略平均57.2%成功率,本方法在六类泛化下仅用一次示范即达87.8%成功率。
原文摘要 · Abstract (English)
Visuomotor policies learned from teleoperated demonstrations face challenges such as lengthy data collection, high costs, and limited data diversity. Existing approaches address these issues by augmenting image observations in RGB space or employing Real-to-Sim-to-Real pipelines based on physical simulators. However, the former is constrained to 2D data augmentation, while the latter suffers from imprecise physical simulation caused by inaccurate geometric reconstruction. This paper introduces RoboSplat, a novel method that generates diverse, visually realistic demonstrations by directly manipulating 3D Gaussians. Specifically, we reconstruct the scene through 3D Gaussian Splatting (3DGS), directly edit the reconstructed scene, and augment data across six types of generalization with five techniques: 3D Gaussian replacement for varying object types, scene appearance, and robot embodiments; equivariant transformations for different object poses; visual attribute editing for various lighting conditions; novel view synthesis for new camera perspectives; and 3D content generation for diverse object types. Comprehensive real-world experiments demonstrate that RoboSplat significantly enhances the generalization of visuomotor policies under diverse disturbances. Notably, while policies trained on hundreds of real-world demonstrations with additional 2D data augmentation achieve an average success rate of 57.2%, RoboSplat attains 87.8% in one-shot settings across six types of generalization in the real world.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。