用真实场景生成高保真仿真数据，让机器人在仿真中训练后直接在现实任务中成功。
Re$^3$Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation
- 通过3D重建与神经渲染还原真实场景，实现仿真中实时跨视角渲染。
- 仅用仿真数据训练的策略在真实世界零样本迁移成功率超58%。
- 适合需要低成本数据集构建的机器人操控研究者使用。
机器人真实世界数据采集成本高、耗时长，需专业人员与昂贵设备；而现有仿真存在几何与视觉差异，导致仿真实现不了真实泛化。为此，本文提出一种3D- photorealistic 的真实到仿真系统（RE$^3$SIM），结合先进的3D重建与神经渲染技术，精准复现真实场景，实现在物理引擎中实时渲染多视角仿真摄像头。利用特权信息高效收集仿真中的专家示范，并通过模仿学习训练机器人策略，验证了真实→仿真→真实流程的有效性。值得注意的是，仅依赖仿真数据即可实现零样本的仿真到现实迁移，平均成功率超过58%。为进一步提升真实到仿真能力，我们构建了一个大规模仿真数据集，证明了基于仿真数据训练的鲁棒策略可泛化至多种物体。代码与演示见：https://re3sim.github.io/。
原文摘要 · Abstract (English)
Real-world data collection for robotics is costly and resource-intensive, requiring skilled operators and expensive hardware. Simulations offer a scalable alternative but often fail to achieve sim-to-real generalization due to geometric and visual gaps. To address these challenges, we propose a 3D-photorealistic real-to-sim system, namely, RE$^3$SIM, addressing geometric and visual sim-to-real gaps. RE$^3$SIM employs advanced 3D reconstruction and neural rendering techniques to faithfully recreate real-world scenarios, enabling real-time rendering of simulated cross-view cameras within a physics-based simulator. By utilizing privileged information to collect expert demonstrations efficiently in simulation, and train robot policies with imitation learning, we validate the effectiveness of the real-to-sim-to-real pipeline across various manipulation task scenarios. Notably, with only simulated data, we can achieve zero-shot sim-to-real transfer with an average success rate exceeding 58%. To push the limit of real-to-sim, we further generate a large-scale simulation dataset, demonstrating how a robust policy can be built from simulation data that generalizes across various objects. Codes and demos are available at: https://re3sim.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。