用事件相机提升雨天视频去雨效果,更准更稳。
RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics

- 先分后融:分别提取雨和背景成分再融合
- 事件流专治雨滴运动,抑制干扰噪声
- 适合做自动驾驶等户外视觉系统
视频去雨旨在恢复恶劣天气下的清晰视觉内容,以保障感知可靠性。现有方法主要依赖RGB序列和时间冗余,但在动态雨景中,雨痕、纹理、边界、运动与遮挡常具相似视觉特征,导致恢复模糊。事件相机提供高时间分辨率的运动敏感信息,但事件流也包含传感器噪声和背景触发响应,直接融合易引入跨模态干扰。为此,我们提出RainDancer,一种基于‘先分解后交互’范式的渐进式RGB-事件视频去雨框架。核心思想是在跨模态交互前,分别在各模态内分离雨与背景成分:在RGB分支,帧特征逐步分解为雨与背景表示;在事件分支,设计雨导向的脉冲神经网络模块捕捉与雨运动相关的稀疏突发事件动态。随后,在语义对齐的成分间进行融合,实现结构保留与雨迹抑制。进一步引入事件域监督,规范稀疏事件重建、结构一致性及梯度方向。在合成与真实RGB-事件视频去雨数据集上的实验表明,该方法在定量指标、视觉质量及下游感知鲁棒性上均优于现有方法。代码已开源:https://github.com/AE86-plus/RainDancer。
原文摘要 · Abstract (English)
Video deraining aims to recover clean visual content from rainy videos for reliable perception under adverse weather. Existing methods mainly rely on RGB sequences and temporal redundancy, but RGB-only restoration remains ambiguous in dynamic rainy scenes, where rain streaks, textures, boundaries, motion, and occlusions may share similar visual patterns. Event cameras provide complementary motion-sensitive cues with high temporal resolution, but event streams also contain sensor noise and background-triggered responses, so direct RGB-Event fusion may introduce cross-modal interference. To address this issue, we propose RainDancer, a progressive RGB-Event video deraining framework based on a decompose-before-interact paradigm. The core idea is to separate rain and background components within each modality before cross-modal interaction. In the RGB branch, frame features are progressively decomposed into rain and background representations. In the event branch, a rain-oriented spiking neural network module captures sparse and bursty event dynamics associated with rain motion. Component-level fusion is then performed between semantically aligned representations for structure preservation and rain suppression. We further introduce event-domain supervision to regularize sparse event reconstruction, structural consistency, and gradient orientation. Experiments on synthetic and real RGB-Event video deraining datasets demonstrate superior quantitative performance, visual quality, and downstream perception robustness. Code is available at https://github.com/AE86-plus/RainDancer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。