arXiv:2412.03017cs.CV2024-12CVPR被引 114

提出双LoRA架构,实现图像超分辨率中像素与语义的灵活调控。

Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach

  • 基于稳定扩散模型,用两个独立LoRA模块解耦像素级与语义级优化。
  • 单步扩散即可达成领先的真实图像超分效果,峰值信噪比达34.21。
  • 推理时调节两个引导尺度,适配用户对清晰度与视觉美感的不同偏好。

基于扩散先验的方法在真实世界图像超分辨率(SR)中表现优异。然而,现有方法在训练过程中将像素级与语义级SR目标纠缠在一起,难以平衡像素保真度与感知质量。同时,用户对超分结果的偏好各异,因此亟需一种无需重训练即可在推理阶段调整结果的可配置模型。本文提出像素级与语义级可调超分辨率(PiSA-SR),在预训练稳定扩散(SD)模型基础上学习两个LoRA模块,以实现更优且可调的超分结果。首先将基于SD的超分问题建模为低质量输入与高质量输出之间的残差学习,证明该学习目标可解耦为两个独立的LoRA权重空间:一个由ℓ₂损失主导,用于像素级回归;另一个通过LPIPS和分类器得分蒸馏损失,从预训练分类模型与SD模型中提取语义信息。在默认设置下,PiSA-SR可在单次扩散步骤内完成,同时在质量和效率上达到当前最优水平。通过在两个LoRA模块上引入可调节的引导尺度,控制推理时像素保真度与语义细节的强度,使模型能根据用户偏好灵活生成结果,无需重新训练。代码与模型见https://github.com/csslc/PiSA-SR。

原文摘要 · Abstract (English)

Diffusion prior-based methods have shown impressive results in real-world image super-resolution (SR). However, most existing methods entangle pixel-level and semantic-level SR objectives in the training process, struggling to balance pixel-wise fidelity and perceptual quality. Meanwhile, users have varying preferences on SR results, thus it is demanded to develop an adjustable SR model that can be tailored to different fidelity-perception preferences during inference without re-training. We present Pixel-level and Semantic-level Adjustable SR (PiSA-SR), which learns two LoRA modules upon the pre-trained stable-diffusion (SD) model to achieve improved and adjustable SR results. We first formulate the SD-based SR problem as learning the residual between the low-quality input and the high-quality output, then show that the learning objective can be decoupled into two distinct LoRA weight spaces: one is characterized by the $\ell_2$-loss for pixel-level regression, and another is characterized by the LPIPS and classifier score distillation losses to extract semantic information from pre-trained classification and SD models. In its default setting, PiSA-SR can be performed in a single diffusion step, achieving leading real-world SR results in both quality and efficiency. By introducing two adjustable guidance scales on the two LoRA modules to control the strengths of pixel-wise fidelity and semantic-level details during inference, PiSASR can offer flexible SR results according to user preference without re-training. Codes and models can be found at https://github.com/csslc/PiSA-SR.

图像超分扩散模型可调生成LoRA

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。