arXiv:2409.03455cs.CV2024-09被引 3

用扩散模型生成特定天气图像,无需原始数据即可训练轻量级修复模型。

Data-free Distillation with Degradation-prompt Diffusion for Multi-weather Image Restoration

  • 用预训练扩散模型替代GAN生成伪图像,避免训练不稳定。
  • 通过退化提示适配器生成与天气相关的退化图像,提升内容一致性。
  • 适合资源受限设备上的多天气图像修复任务,无需原始数据集。

多天气图像修复取得了显著进展,但模型容量增大和数据获取成本高昂限制了其在内存有限设备上的应用。数据自由蒸馏提供了一种新路径:无需原始训练数据,即可从预训练教师模型中学习轻量级学生模型。现有方法主要依赖生成对抗网络(GAN)生成伪数据或互联网收集的真实数据,但存在训练不稳或域偏移问题。本文提出一种新型数据自由蒸馏框架D4IR,用于多天气图像修复。该框架以预训练扩散模型替代GAN,避免模型坍塌,并引入退化感知提示适配器,实现内容驱动的条件扩散,生成与特定退化类型相关的图像。具体而言,设计基于对比的退化提示适配器,从网络收集的退化图像中提取退化特征;将未配对的清晰图像扰动至稳定扩散的潜在空间,并结合退化提示生成新的域相关退化图像,用于知识蒸馏。实验表明,本方法性能接近使用原始数据蒸馏的模型,且优于其他主流无监督方法。

原文摘要 · Abstract (English)

Multi-weather image restoration has witnessed incredible progress, while the increasing model capacity and expensive data acquisition impair its applications in memory-limited devices. Data-free distillation provides an alternative for allowing to learn a lightweight student model from a pre-trained teacher model without relying on the original training data. The existing data-free learning methods mainly optimize the models with the pseudo data generated by GANs or the real data collected from the Internet. However, they inevitably suffer from the problems of unstable training or domain shifts with the original data. In this paper, we propose a novel Data-free Distillation with Degradation-prompt Diffusion framework for multi-weather Image Restoration (D4IR). It replaces GANs with pre-trained diffusion models to avoid model collapse and incorporates a degradation-aware prompt adapter to facilitate content-driven conditional diffusion for generating domain-related images. Specifically, a contrast-based degradation prompt adapter is firstly designed to capture degradation-aware prompts from web-collected degraded images. Then, the collected unpaired clean images are perturbed to latent features of stable diffusion, and conditioned with the degradation-aware prompts to synthesize new domain-related degraded images for knowledge distillation. Experiments illustrate that our proposal achieves comparable performance to the model distilled with original training data, and is even superior to other mainstream unsupervised methods.

图像修复扩散模型数据自由蒸馏多天气

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。