提出动态堆叠滤波的半监督视频去雨模型,提升真实场景泛化能力。
Semi-Supervised State-Space Model with Dynamic Stacking Filter for Real-World Video Deraining
- 双分支时空状态空间模型,分离处理空间特征与帧间时序依赖。
- 设计动态堆叠滤波器,自适应优化像素级特征,提升去雨精度。
- 引入中值堆叠损失实现半监督学习,适用于真实雨天目标检测任务。
过去十年,深度学习推动了雨天视频修复的显著进展。然而,依赖成对数据的方法在真实场景中泛化能力差,主要因合成雨与真实雨效应差异。为此,我们提出一种双分支时空状态空间模型,以增强视频序列中的雨条去除效果。具体地,设计空间与时间状态空间层,分别提取空间特征并建模帧间时序依赖。为改善多帧特征融合,提出动态堆叠滤波器,自适应逼近统计滤波器,实现更优的像素级特征细化。此外,设计中值堆叠损失,基于雨的稀疏先验生成伪干净块,支持半监督学习。为进一步探索去雨模型在雨天视觉任务中的潜力,构建了一个聚焦雨天目标检测与跟踪的新真实世界基准。方法在多个含大量合成与真实雨天视频的基准上评估,持续在定量指标、视觉质量、效率及下游任务表现上优于现有方法。
原文摘要 · Abstract (English)
Significant progress has been made in video restoration under rainy conditions over the past decade, largely propelled by advancements in deep learning. Nevertheless, existing methods that depend on paired data struggle to generalize effectively to real-world scenarios, primarily due to the disparity between synthetic and authentic rain effects. To address these limitations, we propose a dual-branch spatio-temporal state-space model to enhance rain streak removal in video sequences. Specifically, we design spatial and temporal state-space model layers to extract spatial features and incorporate temporal dependencies across frames, respectively. To improve multi-frame feature fusion, we derive a dynamic stacking filter, which adaptively approximates statistical filters for superior pixel-wise feature refinement. Moreover, we develop a median stacking loss to enable semi-supervised learning by generating pseudo-clean patches based on the sparsity prior of rain. To further explore the capacity of deraining models in supporting other vision-based tasks in rainy environments, we introduce a novel real-world benchmark focused on object detection and tracking in rainy conditions. Our method is extensively evaluated across multiple benchmarks containing numerous synthetic and real-world rainy videos, consistently demonstrating its superiority in quantitative metrics, visual quality, efficiency, and its utility for downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。