构建60万帧夜间雨天视频数据集，提升真实场景去雨效果。
UENR-600K: A Large-Scale Physically Grounded Dataset for Nighttime Video Deraining
- 用3D粒子模拟真实雨滴与灯光交互，生成高保真夜间雨视频。
- 基于该数据集的模型在真实夜雨视频上性能显著超越以往方法。
- 适合做夜间视频去雨、物理仿真生成的研究者使用。
夜间视频去雨因雨滴与人工光源相互作用而尤为困难。与白天均匀雨水不同，夜间雨滴呈现多种颜色且局部受光。现有小规模合成数据集依赖2D雨层叠加，无法捕捉这些物理特性，导致模型难以泛化至真实场景。同时，真实配对夜间视频难以采集，因雨效无法与传感器噪声等其他退化分离。为此，我们提出UENR-600K，一个大规模、物理真实的夜间视频去雨数据集，包含60万帧1080p图像对。利用Unreal Engine模拟雨滴作为3D粒子，确保光照折射、遮挡、雨帘等真实细节。基于此高质量数据，我们通过适配Wan 2.2视频生成模型建立新基准，将去雨视为视频到视频生成任务，利用强生成先验几乎完全弥合仿真到真实差距。大量实验证明，训练于本数据集的模型在真实夜间视频上泛化能力显著增强。
原文摘要 · Abstract (English)
Nighttime video deraining is uniquely challenging because raindrops interact with artificial lighting. Unlike daytime white rain, nighttime rain takes on various colors and appears locally illuminated. Existing small-scale synthetic datasets rely on 2D rain overlays and fail to capture these physical properties, causing models to generalize poorly to real-world night rain. Meanwhile, capturing real paired nighttime videos remains impractical because rain effects cannot be isolated from other degradations like sensor noise. To bridge this gap, we introduce UENR-600K, a large-scale, physically grounded dataset containing 600,000 1080p frame pairs. We utilize Unreal Engine to simulate rain as 3D particles within virtual environments. This approach guarantees photorealism and physically real raindrops, capturing correct details like color refractions, scene occlusions, rain curtains. Leveraging this high-quality data, we establish a new state-of-the-art baseline by adapting the Wan 2.2 video generation model. Our baseline treat deraining as a video-to-video generation task, exploiting strong generative priors to almost entirely bridge the sim-to-real gap. Extensive benchmarking demonstrates that models trained on our dataset generalize significantly better to real-world videos. Project page: https://showlab.github.io/UENR-600K/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。