让机器学会识别3D场景间的空间关系,实现跨场景的路径与物体布局迁移。
Learning 3D Scene Analogies with Neural Contextual Scene Maps
- 通过神经上下文场景图提取语义几何特征,粗到细对齐生成平滑场景映射
- 在多样室内场景中成功迁移轨迹和物体摆放,鲁棒性强于依赖点特征的方法
- 适用于机器人导航、AR/VR路径转移及模仿学习中的长程示范迁移
理解场景上下文对机器在未见或噪声环境中完成任务、迁移先验知识至关重要。由于数据驱动方法难以全面涵盖多样的布局与开放空间,我们提出教会机器识别3D空间中的关系共性。不同于点级或物体级表示,本文引入3D场景类比——即平滑连接3D场景区域的映射,对齐空间关系。相比已研究的单实例级映射,这类场景级映射能连接大范围区域,潜在应用于AR/VR中的轨迹转移、模仿学习中的长示范迁移以及上下文感知的物体重排。为此,我们提出神经上下文场景图,通过提取描述符场总结语义与几何上下文,并以粗到细的方式整体对齐以估计地图。该方法降低对个别特征点的依赖,增强对输入噪声或形状变化的鲁棒性。实验表明,该方法在识别场景类比及在多样室内场景中迁移轨迹或物体位置方面有效,展现出在机器人与AR/VR应用中的潜力。项目页面及代码可通过 https://82magnolia.github.io/3d_scene_analogies/ 获取。
原文摘要 · Abstract (English)
Understanding scene contexts is crucial for machines to perform tasks and adapt prior knowledge in unseen or noisy 3D environments. As data-driven learning is intractable to comprehensively encapsulate diverse ranges of layouts and open spaces, we propose teaching machines to identify relational commonalities in 3D spaces. Instead of focusing on point-wise or object-wise representations, we introduce 3D scene analogies, which are smooth maps between 3D scene regions that align spatial relationships. Unlike well-studied single instance-level maps, these scene-level maps smoothly link large scene regions, potentially enabling unique applications in trajectory transfer in AR/VR, long demonstration transfer for imitation learning, and context-aware object rearrangement. To find 3D scene analogies, we propose neural contextual scene maps, which extract descriptor fields summarizing semantic and geometric contexts, and holistically align them in a coarse-to-fine manner for map estimation. This approach reduces reliance on individual feature points, making it robust to input noise or shape variations. Experiments demonstrate the effectiveness of our approach in identifying scene analogies and transferring trajectories or object placements in diverse indoor scenes, indicating its potential for robotics and AR/VR applications. Project page including the code is available through this link: https://82magnolia.github.io/3d_scene_analogies/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。