用事件相机提升动态场景下深度补全的精度与稳定性。
Event-Driven Dynamic Scene Depth Completion
- 基于事件流动态调节特征对齐与深度滤波,适应快速运动。
- 在真实与合成数据集上显著优于现有方法,尤其在运动区域表现更好。
- 首个事件驱动深度补全框架,适合做动态环境感知的研究者。
动态场景中的深度补全因自身运动和物体运动导致输入模态(如RGB图像和激光雷达)质量下降而面临严峻挑战。传统RGB-D传感器在高速运动下难以实现精确对齐与可靠深度获取。相比之下,事件相机具有高时间分辨率和像素级运动敏感性,能提供互补信息。为此,我们提出首个事件驱动的深度补全框架EventDC,包含两个核心模块:事件调制对齐(EMA)和局部深度滤波(LDF)。在编码器中,EMA利用事件流调制RGB-D特征的采样位置,实现像素重分布以改善对齐与融合;在解码器中,LDF通过事件学习运动感知掩码,优化运动物体附近的深度估计。此外,框架引入两项损失项,分别促进全局对齐与局部深度恢复。我们还建立了首个基于事件的深度补全基准,包含一个真实世界数据集和两个合成数据集,以推动后续研究。大量实验表明,EventDC在该基准上显著优于现有方法。
原文摘要 · Abstract (English)
Depth completion in dynamic scenes poses significant challenges due to rapid ego-motion and object motion, which can severely degrade the quality of input modalities such as RGB images and LiDAR measurements. Conventional RGB-D sensors often struggle to align precisely and capture reliable depth under such conditions. In contrast, event cameras with their high temporal resolution and sensitivity to motion at the pixel level provide complementary cues that are %particularly beneficial in dynamic environments.To this end, we propose EventDC, the first event-driven depth completion framework. It consists of two key components: Event-Modulated Alignment (EMA) and Local Depth Filtering (LDF). Both modules adaptively learn the two fundamental components of convolution operations: offsets and weights conditioned on motion-sensitive event streams. In the encoder, EMA leverages events to modulate the sampling positions of RGB-D features to achieve pixel redistribution for improved alignment and fusion. In the decoder, LDF refines depth estimations around moving objects by learning motion-aware masks from events. Additionally, EventDC incorporates two loss terms to further benefit global alignment and enhance local depth recovery. Moreover, we establish the first benchmark for event-based depth completion comprising one real-world and two synthetic datasets to facilitate future research. Extensive experiments on this benchmark demonstrate the superiority of our EventDC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。