用手机拍的场景重建3D,让AI导航模型更适应真实环境。
EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Device
- 用iPhone拍摄+高斯点云重建真实场景,低成本实现逼真仿真
- 在真实任务中成功率比基线高20%~40%,模拟与现实相关性达0.87-0.97
- 适合想快速部署个性化导航系统的开发者或研究者
Embodied AI 主要依赖仿真训练与评估,但多数环境或为合成场景缺乏真实感,或需昂贵设备重建。本文提出 EmbodiedSplat,通过手机拍摄的部署环境,结合3D高斯点云(GS)技术与Habitat-Sim仿真器,高效重建接近真实条件的训练场景。我们分析了训练策略、预训练数据集及网格重建方法对模拟到现实迁移性能的影响。实验表明,经EmbodiedSplat微调的智能体在真实世界的图像导航任务中,相比在大规模真实数据集(HM3D)和合成数据集(HSSD)上预训练的零样本基线,成功率分别提升20%和40%。重建网格的模拟与现实相关性高达0.87–0.97,证明该方法可低投入适配多样环境。
原文摘要 · Abstract (English)
The field of Embodied AI predominantly relies on simulation for training and evaluation, often using either fully synthetic environments that lack photorealism or high-fidelity real-world reconstructions captured with expensive hardware. As a result, sim-to-real transfer remains a major challenge. In this paper, we introduce EmbodiedSplat, a novel approach that personalizes policy training by efficiently capturing the deployment environment and fine-tuning policies within the reconstructed scenes. Our method leverages 3D Gaussian Splatting (GS) and the Habitat-Sim simulator to bridge the gap between realistic scene capture and effective training environments. Using iPhone-captured deployment scenes, we reconstruct meshes via GS, enabling training in settings that closely approximate real-world conditions. We conduct a comprehensive analysis of training strategies, pre-training datasets, and mesh reconstruction techniques, evaluating their impact on sim-to-real predictivity in real-world scenarios. Experimental results demonstrate that agents fine-tuned with EmbodiedSplat outperform both zero-shot baselines pre-trained on large-scale real-world datasets (HM3D) and synthetically generated datasets (HSSD), achieving absolute success rate improvements of 20% and 40% on real-world Image Navigation task. Moreover, our approach yields a high sim-vs-real correlation (0.87-0.97) for the reconstructed meshes, underscoring its effectiveness in adapting policies to diverse environments with minimal effort. Project page: https://gchhablani.github.io/embodied-splat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。