用扩散模型一键生成或移除视频中的雨雪雾等天气效果，还能精细调节强度。
Controllable Weather Synthesis and Removal with Video Diffusion Models
- 基于扩散模型直接合成雨雪雾云等天气，无需3D建模。
- 在真实视频上实现高质量、保场景的天气效果生成与去除。
- 创新数据策略解决配对数据少问题，适合影视后期与自动驾驶测试。
在视频中生成逼真且可控的天气效果对诸多应用具有价值。基于物理的天气模拟需要精确重建，难以扩展到真实世界视频；而现有视频编辑方法常缺乏真实感和控制力。本文提出WeatherWeaver，一种视频扩散模型，可直接将雨、雪、雾、云等多种天气效果合成至任意输入视频，无需3D建模。该模型能精确控制天气强度，并支持多种天气类型混合，兼顾真实感与适应性。针对配对训练数据稀缺的问题，提出结合合成视频、生成式图像编辑和自动标注的真实视频的新数据策略。大量实验表明，本方法在天气模拟与去除任务上优于当前最优方法，在多种真实视频上均获得高质量、物理合理且保留场景身份的结果。
原文摘要 · Abstract (English)
Generating realistic and controllable weather effects in videos is valuable for many applications. Physics-based weather simulation requires precise reconstructions that are hard to scale to in-the-wild videos, while current video editing often lacks realism and control. In this work, we introduce WeatherWeaver, a video diffusion model that synthesizes diverse weather effects -- including rain, snow, fog, and clouds -- directly into any input video without the need for 3D modeling. Our model provides precise control over weather effect intensity and supports blending various weather types, ensuring both realism and adaptability. To overcome the scarcity of paired training data, we propose a novel data strategy combining synthetic videos, generative image editing, and auto-labeled real-world videos. Extensive evaluations show that our method outperforms state-of-the-art methods in weather simulation and removal, providing high-quality, physically plausible, and scene-identity-preserving results over various real-world videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。