arXiv:2608.02953cs.CV2026-08

用真实视频训练,让自动驾驶天气转换更逼真且场景不变形。

RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models

论文配图:RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models
图 1 · 摘自论文原文
  • 从真实视频中学习天气变化规律,逐步提升生成质量。
  • 在多个测试集上实现更高视觉真实度和结构一致性。
  • 适合自动驾驶系统开发与跨天气泛化能力研究者。

真实天气转换对自动驾驶系统开发与评估至关重要,但大规模采集同一场景在不同天气下的配对视频不现实。现有方法依赖合成数据、3D天气编辑或几何条件生成,常牺牲天气真实感或场景保真度。本文提出 RealWeather,一种面向真实且场景忠实的驾驶世界模型。核心思想是直接从真实视频中学习真实的天气动态。具体采用渐进式真实感自举(Progressive Realism Bootstrapping)策略:初始阶段使用伪风格条件视频训练,随着训练推进,逐步替换为模型自身生成的越来越真实的视频。该策略弥合了伪-真实域差距,使模型能自然适应真实输入分布,并支持双向清晰-恶劣天气转换。为进一步确保结构完整性、抑制幻觉,引入场景保真度强化学习优化(Scene-Fidelity RL Optimization),通过奖励驱动策略显式惩罚对关键驾驶要素的改动。大量实验表明,RealWeather 在视觉真实度和结构保持方面显著优于现有方法,同时可生成长尾天气场景,具备强零样本分布外泛化能力。视频演示见 https://hust-umi.github.io/RealWeather/。

原文摘要 · Abstract (English)

Realistic weather translation is valuable for developing and evaluating autonomous driving systems, yet collecting paired videos of the same scenes under different weather conditions at scale is impractical. Existing methods therefore rely on synthetic data, 3D weather editing, or geometry-conditioned generation, often compromising weather realism or scene fidelity. We propose RealWeather, a driving world model for both realistic and scene-faithful weather translation. Our key idea is to learn authentic weather dynamics directly from real-world videos. Specifically, RealWeather employs Progressive Realism Bootstrapping, an iterative data-refinement strategy. Assisted by an auxiliary Pseudo-Clear Generation pipeline, training initially starts with pseudo-style conditioning videos. As training proceeds, these inputs are progressively replaced with increasingly realistic videos generated by the model itself. This strategy bridges the pseudo-to-real domain gap, allowing the model to adapt seamlessly to real-world input distributions and naturally support bidirectional clear adverse translation. Furthermore, to strictly enforce structural integrity and suppress hallucinations, we introduce Scene-Fidelity RL Optimization, a reward-driven policy optimization strategy that explicitly penalizes alterations to safety-critical driving elements. Extensive experiments demonstrate that RealWeather significantly outperforms existing methods in visual realism and structural preservation, while enabling robust long-tail weather scenario generation and strong zero-shot out-of-distribution generalization. Our video demos can be found at https://hust-umi.github.io/RealWeather/.

自动驾驶天气转换世界模型场景保真

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。