让城市场景随心变天气,还能控制雨雪大小和物体动作。
WeatherCity: Urban Scene Reconstruction with Controllable Multi-Weather Transformation
- 用文本引导图像编辑+天气专属解码器,实现多天气建模。
- 支持轻度降雨到大雪等精细天气控制,时序一致且画质高。
- 适合自动驾驶仿真、虚拟拍摄,需3D重建与可控生成的场景
可编辑的高保真4D场景对自动驾驶至关重要,可用于端到端训练与闭环仿真。然而现有重建方法多局限于复现观测场景,缺乏多样天气模拟能力;而图像级天气编辑易引入场景伪影,且控制性差。为此,我们提出WeatherCity框架,通过文本引导图像编辑模型实现背景天气灵活调整;针对多天气建模难题,设计基于共享场景特征与专用天气解码器的天气高斯表示,并引入内容一致性优化以保证不同天气下的一致性;此外,构建物理驱动模型,通过粒子与运动模式模拟动态天气效果。在多个数据集和场景上的实验表明,WeatherCity在4D重建与天气编辑中兼具灵活性、高保真度与时间一致性。该框架不仅能精细控制天气(如轻雨、大雪),还支持场景内物体级操作。代码已开源。
原文摘要 · Abstract (English)
Editable high-fidelity 4D scenes are crucial for autonomous driving, as they can be applied to end-to-end training and closed-loop simulation. However, existing reconstruction methods are primarily limited to replicating observed scenes and lack the capability for diverse weather simulation. While image-level weather editing methods tend to introduce scene artifacts and offer poor controllability over the weather effects. To address these limitations, we propose \textbf{WeatherCity}, a novel framework for 4D urban scene reconstruction and weather editing. Specifically, we leverage a text-guided image editing model to achieve flexible editing of image weather backgrounds. To tackle the challenge of multi-weather modeling, we introduce a novel weather Gaussian representation based on shared scene features and dedicated weather-specific decoders. This representation is further enhanced with a content consistency optimization, ensuring coherent modeling across different weather conditions. Additionally, we design a physics-driven model that simulates dynamic weather effects through particles and motion patterns. Extensive experiments on multiple datasets and various scenes demonstrate that WeatherCity achieves flexible controllability, high fidelity, and temporal consistency in 4D reconstruction and weather editing. Our framework not only enables fine-grained control over weather conditions (e.g., light rain and heavy snow) but also supports object-level manipulation within the scene. Codes are released at https://github.com/IRMVLab/WeatherCity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。