用语义特征区分提升图像超分真实感,还能无参考评估画质。
Exploring Semantic Feature Discrimination for Perceptual Image Super-Resolution and Opinion-Unaware No-Reference Image Quality Assessment
- 通过CLIP提取语义特征,设计像素级与文本引导的双重判别器
- 在经典与真实场景超分任务上均显著提升视觉质量
- 无需额外训练即可实现高精度无参考画质评估,适合工业部署
生成对抗网络(GAN)广泛应用于图像超分辨率(SR)以提升感知质量。然而,现有基于GAN的SR方法通常对图像进行粗粒度判别,忽略图像语义信息,导致超分网络难以学习到细粒度且与语义相关的纹理细节。为此,我们提出一种语义特征判别方法(SFD)用于感知超分辨率。具体地,首先设计一个特征判别器(Feat-D),对CLIP提取的像素级中层语义特征进行判别,使超分图像的特征分布与高质量图像对齐。此外,引入可学习提示对(LPP),通过对抗方式在CLIP更抽象的输出特征上进行文本引导判别(TG-D),进一步增强判别能力。结合Feat-D与TG-D,SFD能有效区分低质量与高质量图像的语义特征分布,促使超分网络生成更真实、语义相关的纹理。此外,基于训练好的Feat-D与LPP,我们提出一种新型无参考画质评估方法(SFD-IQA),无需额外训练即可显著提升无参考画质评估性能。在经典图像超分、真实世界超分及无参考画质评估任务上的大量实验验证了所提方法的有效性。
原文摘要 · Abstract (English)
Generative Adversarial Networks (GANs) have been widely applied to image super-resolution (SR) to enhance the perceptual quality. However, most existing GAN-based SR methods typically perform coarse-grained discrimination directly on images and ignore the semantic information of images, making it challenging for the super resolution networks (SRN) to learn fine-grained and semantic-related texture details. To alleviate this issue, we propose a semantic feature discrimination method, SFD, for perceptual SR. Specifically, we first design a feature discriminator (Feat-D), to discriminate the pixel-wise middle semantic features from CLIP, aligning the feature distributions of SR images with that of high-quality images. Additionally, we propose a text-guided discrimination method (TG-D) by introducing learnable prompt pairs (LPP) in an adversarial manner to perform discrimination on the more abstract output feature of CLIP, further enhancing the discriminative ability of our method. With both Feat-D and TG-D, our SFD can effectively distinguish between the semantic feature distributions of low-quality and high-quality images, encouraging SRN to generate more realistic and semantic-relevant textures. Furthermore, based on the trained Feat-D and LPP, we propose a novel opinion-unaware no-reference image quality assessment (OU NR-IQA) method, SFD-IQA, greatly improving OU NR-IQA performance without any additional targeted training. Extensive experiments on classical SISR, real-world SISR, and OU NR-IQA tasks demonstrate the effectiveness of our proposed methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。