通过内在空间控制天气编辑，实现更精准的图像风格转换。
IntrinsicWeather: Controllable Weather Editing in Intrinsic Space
- 基于扩散模型构建逆向与正向渲染器，从图像生成材质、几何和光照图。
- 在38k合成与18k真实图像上实现细粒度天气控制，优于现有方法。
- 适合自动驾驶等需鲁棒感知的场景，提升恶劣天气下检测精度。
我们提出IntrinsicWeather，一种基于扩散模型的内在空间可控天气编辑框架。该框架包含两个基于扩散先验的组件：逆向渲染器从输入图像估计材质属性、场景几何和光照，生成内在图；正向渲染器利用这些几何与材质图，结合描述特定天气条件的文本提示，生成最终图像。内在图相比传统像素空间编辑提升了可控性。我们设计了内在图感知注意力机制，增强大场景中空间对应与分解质量。正向渲染中，通过CLIP空间插值天气提示实现细粒度控制。此外，我们构建了包含38,000张合成图像和18,000张真实图像的数据集，每张均标注内在图。IntrinsicWeather在性能上超越主流像素空间编辑、天气复原及基于渲染的方法，在自动驾驶等下游任务中展现潜力，可提升复杂天气下检测与分割的鲁棒性。
原文摘要 · Abstract (English)
We present IntrinsicWeather, a diffusion-based framework for controllable weather editing in intrinsic space. Our framework includes two components based on diffusion priors: an inverse renderer that estimates material properties, scene geometry, and lighting as intrinsic maps from an input image, and a forward renderer that utilizes these geometry and material maps along with a text prompt that describes specific weather conditions to generate a final image. The intrinsic maps enhance controllability compared to traditional pixel-space editing approaches. We propose an intrinsic map-aware attention mechanism that improves spatial correspondence and decomposition quality in large outdoor scenes. For forward rendering, we leverage CLIP-space interpolation of weather prompts to achieve fine-grained weather control. We also introduce a synthetic and a real-world dataset, containing 38k and 18k images under various weather conditions, each with intrinsic map annotations. IntrinsicWeather outperforms state-of-the-art pixel-space editing approaches, weather restoration methods, and rendering-based methods, showing promise for downstream tasks such as autonomous driving, enhancing the robustness of detection and segmentation in challenging weather scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。