arXiv:2506.04120cs.ROcs.GR2025-06被引 6

用机器人数据直接生成高保真物理仿真,无需精确标定

Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot Data

  • 融合3D高斯泼溅与显式物体网格,统一表示场景
  • 端到端优化实现几何、外观、姿态与物理参数联合校准
  • 适用于有遮挡和噪声的现实机器人数据,适合工业部署

从真实机器人运动直接创建精确的物理仿真具有重要价值,但面临遮挡、相机位姿噪声和动态场景元素等挑战,难以构建未见物体的几何准确且逼真的数字孪生。本文提出一种全新的真实到仿真框架,同时应对上述问题。核心思路是将3D高斯泼溅的逼真渲染能力与适合物理仿真的显式物体网格结合,在单一表示中实现融合。我们设计了端到端优化流程,利用可微分渲染与可微分物理(基于MuJoCo),直接从原始且不精确的机器人轨迹中联合优化所有场景成分——包括物体几何、外观、机器人位姿及物理参数。该统一优化使我们能同时实现高保真物体网格重建、生成逼真新视角图像,并完成无标注的机器人位姿校准。我们在模拟环境和使用ALOHA 2双臂机械臂的真实复杂序列上验证了方法的有效性,显著提升了真实到仿真转换的实用性和鲁棒性。

原文摘要 · Abstract (English)

Creating accurate, physical simulations directly from real-world robot motion holds great value for safe, scalable, and affordable robot learning, yet remains exceptionally challenging. Real robot data suffers from occlusions, noisy camera poses, dynamic scene elements, which hinder the creation of geometrically accurate and photorealistic digital twins of unseen objects. We introduce a novel real-to-sim framework tackling all these challenges at once. Our key insight is a hybrid scene representation merging the photorealistic rendering of 3D Gaussian Splatting with explicit object meshes suitable for physics simulation within a single representation. We propose an end-to-end optimization pipeline that leverages differentiable rendering and differentiable physics within MuJoCo to jointly refine all scene components - from object geometry and appearance to robot poses and physical parameters - directly from raw and imprecise robot trajectories. This unified optimization allows us to simultaneously achieve high-fidelity object mesh reconstruction, generate photorealistic novel views, and perform annotation-free robot pose calibration. We demonstrate the effectiveness of our approach both in simulation and on challenging real-world sequences using an ALOHA 2 bi-manual manipulator, enabling more practical and robust real-to-simulation pipelines.

真实到仿真物理仿真机器人学习3D重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。