arXiv:2501.02807cs.CV2025-01AAAI被引 28

提升事件相机在非理想条件下的3D重建能力,支持大场景和不准确位姿。

AE-NeRF: Augmenting Event-Based Neural Radiance Fields for Non-ideal Conditions and Larger Scene

  • 利用事件流密度联合优化位姿与神经辐射场,增强鲁棒性。
  • 在真实与合成数据上实现当前最优的事件相机3D重建效果。
  • 适合需要高动态、低延迟3D重建的实际应用开发者。

与基于帧的方法相比,基于事件相机的类脑计算成像具有运动模糊小、时间分辨率高、动态范围大的优势。将多视角一致性与神经辐射场(NeRF)结合,并利用事件相机的独特优势,推动了从移动事件相机捕获的数据中重建NeRF的研究。尽管表现优异,现有方法依赖于均匀且高质量的事件序列及精确相机位姿,主要关注物体级重建,限制了实际应用。本文提出AE-NeRF,解决在非理想条件下学习事件相机驱动的NeRF所面临的挑战,包括非均匀事件序列、噪声位姿和不同尺度场景。该方法利用事件流密度,联合学习位姿校正模块与事件驱动神经辐射场(e-NeRF)框架,以实现对不准确位姿的鲁棒3D重建。为扩展至大场景,提出分层事件蒸馏机制,结合提案e-NeRF与普通e-NeRF网络,重采样并优化重建过程。此外,引入事件重建损失和时序损失,提升重建场景的视角一致性。我们建立了一个综合性基准,包含大规模场景以模拟实际非理想条件,涵盖合成与具有挑战性的真实事件数据集。实验结果表明,本方法在事件驱动3D重建任务上达到新最优性能。

原文摘要 · Abstract (English)

Compared to frame-based methods, computational neuromorphic imaging using event cameras offers significant advantages, such as minimal motion blur, enhanced temporal resolution, and high dynamic range. The multi-view consistency of Neural Radiance Fields combined with the unique benefits of event cameras, has spurred recent research into reconstructing NeRF from data captured by moving event cameras. While showing impressive performance, existing methods rely on ideal conditions with the availability of uniform and high-quality event sequences and accurate camera poses, and mainly focus on the object level reconstruction, thus limiting their practical applications. In this work, we propose AE-NeRF to address the challenges of learning event-based NeRF from non-ideal conditions, including non-uniform event sequences, noisy poses, and various scales of scenes. Our method exploits the density of event streams and jointly learn a pose correction module with an event-based NeRF (e-NeRF) framework for robust 3D reconstruction from inaccurate camera poses. To generalize to larger scenes, we propose hierarchical event distillation with a proposal e-NeRF network and a vanilla e-NeRF network to resample and refine the reconstruction process. We further propose an event reconstruction loss and a temporal loss to improve the view consistency of the reconstructed scene. We established a comprehensive benchmark that includes large-scale scenes to simulate practical non-ideal conditions, incorporating both synthetic and challenging real-world event datasets. The experimental results show that our method achieves a new state-of-the-art in event-based 3D reconstruction.

事件相机3D重建神经辐射场鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。