用CLIP自动识别雨型,动态分配专用网络处理不同降雨
CLIP-driven rain perception: Adaptive deraining with pattern-aware network routing and mask-guided cross-attention
- 通过CLIP计算图文匹配度,自动感知雨滴密度、方向等模式
- 在多个数据集上达到最优效果,尤其在复杂混合数据表现突出
- 适合需要精准处理多样降雨场景的图像去雨任务
现有去雨模型对所有含雨图像使用单一网络处理,但不同雨型差异显著,单一网络难以应对多种雨滴密度、条纹方向和雨强。为此,我们提出一种基于CLIP的雨型感知网络(CLIP-RPN),利用CLIP通过计算视觉-语言匹配分数自动感知雨型,并自适应地将输入路由至对应子网络处理,如不同雨滴密度、条纹方向和降雨强度。CLIP-RPN借助CLIP的跨模态对齐能力实现语义感知的雨型识别,驱动动态子网络路由机制,根据检测到的雨型激活专用处理分支,显著提升模型对多样化降雨条件的处理能力。此外,在子网络中引入多尺度雨掩码预测的遮罩引导交叉注意力(MGCA)机制,通过交叉注意力增强雨区与干净背景间的上下文交互。还设计了动态损失调度机制(DLS),自适应调整优化过程中的梯度,相比常用的$l_1$或$l_2$损失更契合训练动态,取得更好效果。方法在多个数据集上达到当前最优性能,尤其在复杂混合数据集表现优异。
原文摘要 · Abstract (English)
Existing deraining models process all rainy images within a single network. However, different rain patterns have significant variations, which makes it challenging for a single network to handle diverse types of raindrops and streaks. To address this limitation, we propose a novel CLIP-driven rain perception network (CLIP-RPN) that leverages CLIP to automatically perceive rain patterns by computing visual-language matching scores and adaptively routing to sub-networks to handle different rain patterns, such as varying raindrop densities, streak orientations, and rainfall intensity. CLIP-RPN establishes semantic-aware rain pattern recognition through CLIP's cross-modal visual-language alignment capabilities, enabling automatic identification of precipitation characteristics across different rain scenarios. This rain pattern awareness drives an adaptive subnetwork routing mechanism where specialized processing branches are dynamically activated based on the detected rain type, significantly enhancing the model's capacity to handle diverse rainfall conditions. Furthermore, within sub-networks of CLIP-RPN, we introduce a mask-guided cross-attention mechanism (MGCA) that predicts precise rain masks at multi-scale to facilitate contextual interactions between rainy regions and clean background areas by cross-attention. We also introduces a dynamic loss scheduling mechanism (DLS) to adaptively adjust the gradients for the optimization process of CLIP-RPN. Compared with the commonly used $l_1$ or $l_2$ loss, DLS is more compatible with the inherent dynamics of the network training process, thus achieving enhanced outcomes. Our method achieves state-of-the-art performance across multiple datasets, particularly excelling in complex mixed datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。