用可学习提示增强扩散模型,一次修复雪、雾、雨多种天气退化。
DA2Diff: Exploring Degradation-aware Adaptive Diffusion Priors for All-in-One Weather Restoration
- 通过CLIP空间中的提示-图像相似性约束,学习针对不同天气的可调特征。
- 在多个真实数据集上优于现有方法,尤其对复杂混合退化效果更佳。
- 适合需要统一处理多种天气退化的实际视觉系统开发者。
恶劣天气下的图像恢复是众多视觉应用的关键任务。近年来,能统一处理多种天气退化的端到端框架展现出潜力,但不同天气导致的退化模式多样,且现实中的退化复杂多变,仍具挑战。为此,我们提出一种新的扩散范式——DA2Diff,引入退化感知自适应先验,实现全场景天气恢复。该方法利用CLIP模型感知退化特征,通过一组可学习提示,在CLIP空间中通过提示-图像相似性约束捕捉雪、雾、雨等不同天气的表征。将这些提示嵌入扩散模型的特定提示引导模块,实现多天气类型修复。为进一步提升对复杂退化的适应能力,设计了动态专家选择调制器,基于动态天气感知路由机制,为每张退化图像灵活调度不同数量的修复专家,使模型能自适应恢复多样化退化。实验结果表明,DA2Diff在定量与定性评估中均优于现有最先进方法。源代码将在论文录用后公开。
原文摘要 · Abstract (English)
Image restoration under adverse weather conditions is a critical task for many vision-based applications. Recent all-in-one frameworks that handle multiple weather degradations within a unified model have shown potential. However, the diversity of degradation patterns across different weather conditions, as well as the complex and varied nature of real-world degradations, pose significant challenges for multiple weather removal. To address these challenges, we propose an innovative diffusion paradigm with degradation-aware adaptive priors for all-in-one weather restoration, termed DA2Diff. It is a new exploration that applies CLIP to perceive degradation-aware properties for better multi-weather restoration. Specifically, we deploy a set of learnable prompts to capture degradation-aware representations by the prompt-image similarity constraints in the CLIP space. By aligning the snowy/hazy/rainy images with snow/haze/rain prompts, each prompt contributes to different weather degradation characteristics. The learned prompts are then integrated into the diffusion model via the designed weather specific prompt guidance module, making it possible to restore multiple weather types. To further improve the adaptiveness to complex weather degradations, we propose a dynamic expert selection modulator that employs a dynamic weather-aware router to flexibly dispatch varying numbers of restoration experts for each weather-distorted image, allowing the diffusion model to restore diverse degradations adaptively. Experimental results substantiate the favorable performance of DA2Diff over state-of-the-arts in quantitative and qualitative evaluation. Source code will be available after acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。