用神经形态传感器的彩色光流数据,精准去除非均匀雨痕。
SpikeDerain: Unveiling Clear Videos from Rainy Sequences Using Color Spike Streams
- 利用光流事件流建模雨滴动态,实现时序对齐的去雨。
- 合成带参数的连续雨纹数据,解决真实降雨数据稀缺问题。
- 在极端雨况下仍有效,适合高动态场景视频修复。
从带雨痕的视频中恢复清晰画面面临巨大挑战,因雨丝运动迅速,传统帧同步视觉传感器难以准确捕捉。近年来,神经形态传感器以微秒级时间分辨率和高动态范围带来新范式,但现有融合事件流与RGB图像的多模态方法,在真实场景中受硬件同步误差和计算冗余影响,难以处理雨滴复杂的时空干扰。本文提出一种颜色光流事件去雨网络(SpikeDerain),可重建动态场景的光流事件流并精确去除雨痕。为解决真实连续降雨场景的数据稀缺问题,设计了一种物理可解释的雨痕合成模型,基于任意背景图像生成参数化的连续雨纹。实验表明,该网络在合成数据上训练后,仍能在极端降雨条件下保持高度鲁棒性。结果验证了方法在不同雨强和数据集上的有效性与稳健性,为视频去雨任务树立了新标准。代码即将开源。
原文摘要 · Abstract (English)
Restoring clear frames from rainy videos presents a significant challenge due to the rapid motion of rain streaks. Traditional frame-based visual sensors, which capture scene content synchronously, struggle to capture the fast-moving details of rain accurately. In recent years, neuromorphic sensors have introduced a new paradigm for dynamic scene perception, offering microsecond temporal resolution and high dynamic range. However, existing multimodal methods that fuse event streams with RGB images face difficulties in handling the complex spatiotemporal interference of raindrops in real scenes, primarily due to hardware synchronization errors and computational redundancy. In this paper, we propose a Color Spike Stream Deraining Network (SpikeDerain), capable of reconstructing spike streams of dynamic scenes and accurately removing rain streaks. To address the challenges of data scarcity in real continuous rainfall scenes, we design a physically interpretable rain streak synthesis model that generates parameterized continuous rain patterns based on arbitrary background images. Experimental results demonstrate that the network, trained with this synthetic data, remains highly robust even under extreme rainfall conditions. These findings highlight the effectiveness and robustness of our method across varying rainfall levels and datasets, setting new standards for video deraining tasks. The code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。