无需训练即可去除动态场景中的雨痕,利用扩散模型直接修复视频。
Zero-Shot Video Deraining with Video Diffusion Models
- 用负向提示词干预扩散模型的生成过程,实现零样本去雨。
- 在真实雨天视频上表现优于现有方法,尤其对动态背景效果显著。
- 适合缺乏标注数据、需快速部署的视频去雨场景。
现有视频去雨方法多依赖成对数据集进行训练,合成数据限制其在真实雨天中的泛化能力,而静态摄像头采集的数据又难以处理动态场景中的背景与相机运动。近期微调扩散模型虽有进展,但会削弱生成先验,影响对未见情况的泛化。本文提出首个无需训练、不依赖合成数据的零样本视频去雨方法,利用预训练文本到视频扩散模型的强大泛化能力。通过将输入视频反推至扩散模型隐空间,结合负向提示词干预重建过程,使输出远离模型对雨的固有概念。核心创新是注意力切换机制,有效保留动态背景和输入与去雨视频间的结构一致性,缓解了简单负向提示带来的伪影。在真实雨天视频数据集上大量实验表明,该方法显著优于现有技术,且无需监督训练即具备强泛化能力。
原文摘要 · Abstract (English)
Existing video deraining methods are often trained on paired datasets, either synthetic, which limits their ability to generalize to real-world rain, or captured by static cameras, which restricts their effectiveness in dynamic scenes with background and camera motion. Furthermore, recent works in fine-tuning diffusion models have shown promising results, but the fine-tuning tends to weaken the generative prior, limiting generalization to unseen cases. In this paper, we introduce the first zero-shot video deraining method for complex dynamic scenes that does not require synthetic data nor model fine-tuning, by leveraging a pretrained text-to-video diffusion model that demonstrates strong generalization capabilities. By inverting an input video into the latent space of diffusion models, its reconstruction process can be intervened and pushed away from the model's concept of rain using negative prompting. At the core of our approach is an attention switching mechanism that we found is crucial for maintaining dynamic backgrounds as well as structural consistency between the input and the derained video, mitigating artifacts introduced by naive negative prompting. Our approach is validated through extensive experiments on real-world rain datasets, demonstrating substantial improvements over prior methods and showcasing robust generalization without the need for supervised training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。