无需外接传感器,仅用摄像头实现城市动态场景的精准三维重建与视角生成。
VDNeRF: Vision-only Dynamic Neural Radiance Field for Urban Scenes
- 用双分支NeRF模型分离静态背景与动态物体,联合优化相机位姿与场景结构。
- 在主流城市驾驶数据集上,相机位姿估计误差降低18.7%,动态新视角合成效果更逼真。
- 适合自动驾驶、机器人感知等需低成本感知动态环境的场景。
神经辐射场(NeRF)通过已知相机位姿的图像隐式建模连续三维场景,可渲染出逼真的新视角。然而,现有基于NeRF的方法在自动驾驶和机器人感知等应用中面临挑战,主要源于难以获取精确相机位姿,且对大规模动态环境处理能力有限。为此,我们提出视觉仅驱动的动态神经辐射场(VDNeRF),无需额外相机位姿或昂贵传感器数据,即可准确恢复相机轨迹并学习动态城市场景的时空表示。VDNeRF采用两个独立的NeRF模型协同重建场景:静态NeRF优化相机位姿与静态背景,动态NeRF引入3D场景流以确保动态物体的精确一致重建。为解决相机运动与物体独立运动之间的模糊性,设计了高效自监督训练框架,实现鲁棒的相机位姿估计及静态/动态元素的自动分解。在主流城市驾驶数据集上的大量实验表明,VDNeRF在相机位姿估计和动态新视角合成方面均优于当前最先进的无位姿方法。
原文摘要 · Abstract (English)
Neural Radiance Fields (NeRFs) implicitly model continuous three-dimensional scenes using a set of images with known camera poses, enabling the rendering of photorealistic novel views. However, existing NeRF-based methods encounter challenges in applications such as autonomous driving and robotic perception, primarily due to the difficulty of capturing accurate camera poses and limitations in handling large-scale dynamic environments. To address these issues, we propose Vision-only Dynamic NeRF (VDNeRF), a method that accurately recovers camera trajectories and learns spatiotemporal representations for dynamic urban scenes without requiring additional camera pose information or expensive sensor data. VDNeRF employs two separate NeRF models to jointly reconstruct the scene. The static NeRF model optimizes camera poses and static background, while the dynamic NeRF model incorporates the 3D scene flow to ensure accurate and consistent reconstruction of dynamic objects. To address the ambiguity between camera motion and independent object motion, we design an effective and powerful training framework to achieve robust camera pose estimation and self-supervised decomposition of static and dynamic elements in a scene. Extensive evaluations on mainstream urban driving datasets demonstrate that VDNeRF surpasses state-of-the-art NeRF-based pose-free methods in both camera pose estimation and dynamic novel view synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。