用遮蔽退化分类预训练,让模型学会识别并修复各种图像退化。
Universal Image Restoration Pre-training via Masked Degradation Classification
- 通过遮蔽低质量图像,同时做退化类型分类和图像重建。
- 在5种退化合一任务中提升3.77 dB PSNR,真实场景下失真度降低34.8%。
- 可通用修复未见过的退化类型,适合图像修复、预训练研究者。
本文提出一种掩码退化分类预训练方法(MaskDCPT),用于实现通用图像修复的预训练。不同于传统方法,MaskDCPT以图像退化类型作为极弱监督信号,同时利用图像重建增强性能与鲁棒性。该方法包含编码器与两个解码器:编码器从遮蔽的低质量输入中提取特征;分类解码器据此判断退化类型,重建解码器则恢复对应高质量图像。此设计融合了掩码图像建模与对比学习,生成适用于修复任务的通用表征。实验表明,使用MaskDCPT可显著提升CNN与Transformer在5D全类型修复任务中的表现,最小PSNR提升达3.77 dB,真实退化场景下PIQE下降34.8%。模型还展现出对未见退化类型与等级的强大泛化能力。此外,本文构建并发布了包含250万对样本的UIR-2.5M数据集,覆盖19种退化类型、200+退化等级,含合成与真实数据。代码与模型已开源。
原文摘要 · Abstract (English)
This study introduces a Masked Degradation Classification Pre-Training method (MaskDCPT), designed to facilitate the classification of degradation types in input images, leading to comprehensive image restoration pre-training. Unlike conventional pre-training methods, MaskDCPT uses the degradation type of the image as an extremely weak supervision, while simultaneously leveraging the image reconstruction to enhance performance and robustness. MaskDCPT includes an encoder and two decoders: the encoder extracts features from the masked low-quality input image. The classification decoder uses these features to identify the degradation type, whereas the reconstruction decoder aims to reconstruct a corresponding high-quality image. This design allows the pre-training to benefit from both masked image modeling and contrastive learning, resulting in a generalized representation suited for restoration tasks. Benefit from the straightforward yet potent MaskDCPT, the pre-trained encoder can be used to address universal image restoration and achieve outstanding performance. Implementing MaskDCPT significantly improves performance for both convolution neural networks (CNNs) and Transformers, with a minimum increase in PSNR of 3.77 dB in the 5D all-in-one restoration task and a 34.8% reduction in PIQE compared to baseline in real-world degradation scenarios. It also emergences strong generalization to previously unseen degradation types and levels. In addition, we curate and release the UIR-2.5M dataset, which includes 2.5 million paired restoration samples across 19 degradation types and over 200 degradation levels, incorporating both synthetic and real-world data. The dataset, source code, and models are available at https://github.com/MILab-PKU/MaskDCPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。