arXiv:2412.11609cs.CV2024-12被引 6

用文字和图像协同提升超分辨率,16倍放大也能保持细节和语义一致。

CLIP-SR: Collaborative Linguistic and Image Processing for Super-Resolution

  • 融合文本与图像特征,通过CLIP实现细粒度对齐。
  • 在16×超分下仍能还原清晰纹理,避免模糊与失真。
  • 支持文本控制的图像增强,适合需要精准编辑的场景。

卷积神经网络(CNN)虽显著推动了图像超分辨率(SR)发展,但多数方法仅依赖像素级变换,常导致伪影与模糊,尤其在严重降采样(如8×或16×)时更为明显。近期文本引导的SR方法利用文本描述增强细节恢复能力,但往往难以有效对齐,引发语义不一致。为此,我们提出一种多模态语义增强框架,将文本语义与视觉特征融合,有效缓解高度退化低分辨率(LR)图像中的语义错配与细节丢失问题。该方法可在最大16×放大率下实现真实感、高质量的超分辨率重建。框架通过提示预测器、文本-图像融合模块(TIFBlock)与迭代精炼模块,结合对比语言-图像预训练(CLIP)特征,实现渐进式、细粒度对齐的增强过程。这种协同机制使输出图像具备锐利纹理与强语义一致性,即使在大尺度缩放下亦表现优异。大量对比实验与消融研究验证了方法有效性。此外,借助文本语义,该方法还具备超分辨率可编辑性,可在保持语义一致的前提下实现可控增强。代码已公开于https://github.com/hengliusky/CLIP-SR。

原文摘要 · Abstract (English)

Convolutional Neural Networks (CNNs) have significantly advanced Image Super-Resolution (SR), yet most CNN-based methods rely solely on pixel-based transformations, often leading to artifacts and blurring, particularly under severe downsampling rates (\eg, 8$\times$ or 16$\times$). The recently developed text-guided SR approaches leverage textual descriptions to enhance their detail restoration capabilities but frequently struggle with effectively performing alignment, resulting in semantic inconsistencies. To address these challenges, we propose a multi-modal semantic enhancement framework that integrates textual semantics with visual features, effectively mitigating semantic mismatches and detail losses in highly degraded low-resolution (LR) images. Our method enables realistic, high-quality SR to be performed at large upscaling factors, with a maximum scaling ratio of 16$\times$. The framework integrates both text and image inputs using the prompt predictor, the Text-Image Fusion Block (TIFBlock), and the Iterative Refinement Module, leveraging Contrastive Language-Image Pretraining (CLIP) features to guide a progressive enhancement process with fine-grained alignment. This synergy produces high-resolution outputs with sharp textures and strong semantic coherence, even at substantial scaling factors. Extensive comparative experiments and ablation studies validate the effectiveness of our approach. Furthermore, by leveraging textual semantics, our method offers a degree of super-resolution editability, allowing for controlled enhancements while preserving semantic consistency. The code is available at https://github.com/hengliusky/CLIP-SR.

超分辨率多模态文本引导CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。