用轻量提示词实现多种恶劣天气图像修复,参数少且效果好。
TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal
- 用两阶段训练+可学习软提示,按任务自适应调整模型。
- 仅用275万参数,在多任务上表现优于现有方法。
- 通过低秩分解和对比约束,提升任务间关联建模能力。
恶劣天气下的图像恢复已取得广泛研究,尤其是一体化方法在多任务数据集上表现出色。然而,多数方法为每种退化类型设计专用模块或参数,导致参数开销大,且忽略任务间的关联性。为此,我们提出一种参数高效的全功能图像恢复框架,利用任务感知增强提示来应对各类恶劣天气退化。具体采用两阶段训练:先通过监督学习获取通用恢复知识,再通过可训练软提示适配特定退化。关键在于,以任务感知方式增强提示:使用低秩分解捕捉任务通用与特异性特征,并施加对比约束以更好对齐任务间真实相关性。增强提示不仅提升了模型参数效率,也实现了更精准的任务建模(经t-SNE分析验证)。在多个恢复任务上的实验表明,本方法仅需275万参数即达到卓越性能。
原文摘要 · Abstract (English)
Image restoration under adverse weather conditions has been extensively explored, leading to numerous high-performance methods. In particular, recent advances in All-in-One approaches have shown impressive results by training on multi-task image restoration datasets. However, most of these methods rely on dedicated network modules or parameters for each specific degradation type, resulting in a significant parameter overhead. Moreover, the relatedness across different restoration tasks is often overlooked. In light of these issues, we propose a parameter-efficient All-in-One image restoration framework that leverages task-aware enhanced prompts to tackle various adverse weather degradations.Specifically, we adopt a two-stage training paradigm consisting of a pretraining phase and a prompt-tuning phase to mitigate parameter conflicts across tasks. We first employ supervised learning to acquire general restoration knowledge, and then adapt the model to handle specific degradation via trainable soft prompts. Crucially, we enhance these task-specific prompts in a task-aware manner. We apply low-rank decomposition to these prompts to capture both task-general and task-specific characteristics, and impose contrastive constraints to better align them with the actual inter-task relatedness. These enhanced prompts not only improve the parameter efficiency of the restoration model but also enable more accurate task modeling, as evidenced by t-SNE analysis. Experimental results on different restoration tasks demonstrate that the proposed method achieves superior performance with only 2.75M parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。