用事件相机和稀疏RGB帧重建动态场景，解决低光与高速运动难题。
Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streams
- 分段训练时序条件的NeRF模型，融合事件与RGB信号进行时空重建
- 在低光高速场景下，相比纯RGB方法重建精度显著提升
- 首个多视角事件流重建框架，适合高速动态捕捉研究者使用
体素化动态场景重建是计算机视觉中的重要问题，尤其在低光照和快速运动条件下更具挑战性。传统RGB相机在弱光下需延长曝光时间，导致运动模糊；而事件相机以异步方式记录像素亮度变化，对光照依赖小，更适用于高速运动捕捉。本文提出首个基于稀疏多视角事件流与稀疏RGB帧的时空场景重建方法。通过训练一系列跨衰减的时间条件NeRF模型（每段短录制对应一个），并采用事件与RGB损失及稀疏视图正则化进行监督。构建了一个由六个静态事件相机组成的多视角拍摄系统，采集了包含复杂运动的真实世界事件数据集。实验表明，该方法优于基于RGB的基线模型，达到当前最优性能，并开启了事件驱动多视角重建的新方向。代码与数据已公开于 https://4dqv.mpi-inf.mpg.de/DynEventNeRF/
原文摘要 · Abstract (English)
Volumetric reconstruction of dynamic scenes is an important problem in computer vision. It is especially challenging in poor lighting and with fast motion. This is partly due to limitations of RGB cameras: To capture frames under low lighting, the exposure time needs to be increased, which leads to more motion blur. In contrast, event cameras, which record changes in pixel brightness asynchronously, are much less dependent on lighting, making them more suitable for recording fast motion. We hence propose the first method to spatiotemporally reconstruct a scene from sparse multi-view event streams and sparse RGB frames. We train a sequence of cross-faded time-conditioned NeRF models, one per short recording segment. The individual segments are supervised with a set of event- and RGB-based losses and sparse-view regularisation. We assemble a real-world multi-view camera rig with six static event cameras around the object and record a benchmark multi-view event stream dataset of challenging motions. Our work outperforms RGB-based baselines, producing state-of-the-art results, and opens up the topic of multi-view event-based reconstruction as a new path for fast scene capture beyond RGB cameras. The code and the data are released at https://4dqv.mpi-inf.mpg.de/DynEventNeRF/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。