仅用一次机器人动作，端到端学习可模拟可渲染的物理世界模型。
One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering
- 用可微点云与网格外观场联合表示刚体，支持几何、外观、物理属性同步优化。
- 在真实与模拟环境中仅需一次动作序列，即可训练出可直接用于仿真和渲染的世界模型。
- 适合需要快速适应新环境的机器人任务规划与真实世界部署场景。
从稀疏在线观测中构建预测性世界模型，对机器人在新环境中的任务规划与执行至关重要。然而，现有基于可微编程的方法无法联合优化场景的几何、外观与物理属性。本文提出一种新型刚体表示，结合可微点云几何与网格外观场，实现可微碰撞检测与渲染。融合可微物理模拟器后，仅需一次机器人动作序列的视觉与触觉观测，即可端到端优化世界模型。在模拟与真实环境中的多组世界模型识别任务表明，该方法能从单一动作序列中学习出可直接用于仿真与渲染的完整世界模型。代码与附加视频见项目主页：https://tianyi20.github.io/rigid-world-model.github.io/
原文摘要 · Abstract (English)
Identifying predictive world models for robots in novel environments from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appearance, and physical properties of the scene. In this work, we introduce a novel rigid object representation that allows the joint identification of these properties. Our method employs a novel differentiable point-based geometry representation coupled with a grid-based appearance field, which allows differentiable object collision detection and rendering. Combined with a differentiable physical simulator, we achieve end-to-end optimization of world models, given the sparse visual and tactile observations of a physical motion sequence. Through a series of world model identification tasks in simulated and real environments, we show that our method can learn both simulation- and rendering-ready world models from only one robot action sequence. The code and additional videos are available at our project website: https://tianyi20.github.io/rigid-world-model.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。