用语义2D高斯点云桥接仿真与现实,提升机器人抓取泛化能力
Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting
- 通过语义高斯点云提取物体中心的域不变特征
- 在真实场景中保持高且稳定的任务成功率
- 适合需要强泛化能力的机器人仿真部署
机器人操作中的跨域迁移长期受仿真与现实环境间显著域差距困扰。现有方法如领域随机化、自适应和仿真-现实校准常需大量调参或无法泛化至未见场景。我们观察到:若在仿真中训练策略时使用域不变特征,并在真实部署时以相同特征作为输入,则可有效缩小域差距,显著提升策略泛化性。为此,提出语义2D高斯点云(S2GS),一种新表示方法,能提取以物体为中心、域不变的空间特征。S2GS构建多视角2D语义场,并通过特征级高斯点云投射至统一3D空间。语义过滤机制剔除无关背景内容,确保策略学习输入清晰一致。为评估S2GS有效性,采用扩散策略作为下游学习算法,在ManiSkill仿真环境中进行实验,并部署至真实世界。结果表明,S2GS显著提升仿真到现实的迁移能力,在真实场景中维持高且稳定的任务性能。
原文摘要 · Abstract (English)
Cross-domain transfer in robotic manipulation remains a longstanding challenge due to the significant domain gap between simulated and real-world environments. Existing methods such as domain randomization, adaptation, and sim-real calibration often require extensive tuning or fail to generalize to unseen scenarios. To address this issue, we observe that if domain-invariant features are utilized during policy training in simulation, and the same features can be extracted and provided as the input to policy during real-world deployment, the domain gap can be effectively bridged, leading to significantly improved policy generalization. Accordingly, we propose Semantic 2D Gaussian Splatting (S2GS), a novel representation method that extracts object-centric, domain-invariant spatial features. S2GS constructs multi-view 2D semantic fields and projects them into a unified 3D space via feature-level Gaussian splatting. A semantic filtering mechanism removes irrelevant background content, ensuring clean and consistent inputs for policy learning. To evaluate the effectiveness of S2GS, we adopt Diffusion Policy as the downstream learning algorithm and conduct experiments in the ManiSkill simulation environment, followed by real-world deployment. Results demonstrate that S2GS significantly improves sim-to-real transferability, maintaining high and stable task performance in real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。