无需重训,让预训练扩散模型高效生成高清无畸变图像。
RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation
- 用自注意力机制生成高质量中间图,增强结构一致性
- 分步像素空间上采样,显著减少伪影,生成更清晰图像
- 比现有方法更快更清晰,适合实际应用
尽管潜空间扩散模型(如 Stable Diffusion)旨在生成高分辨率(HR)图像,但在高于训练分辨率的条件下常出现严重结构失真。本文提出 RepLDM,一种无需重训的重编程框架,实现高质量、高效率、高分辨率图像生成。该框架包含两阶段:(i) 注意力引导阶段,通过无需训练的自注意力机制生成更高品质的训练分辨率潜变量表示,提升结构一致性;(ii) 渐进式上采样阶段,在像素空间逐步上采样,缓解潜空间上采样导致的严重伪影。第一阶段的有效初始化使高分辨率去噪仅需更少步骤,大幅提升效率。大量实验表明,RepLDM 在质量与效率上均显著优于当前最优方法,适用于真实场景应用。
原文摘要 · Abstract (English)
While latent diffusion models (LDMs), such as Stable Diffusion, are designed for high-resolution (HR) image generation, they often struggle with significant structural distortions when generating images at resolutions higher than their training one. Instead of relying on extensive retraining, a more resource-efficient approach is to reprogram the pretrained model for HR image generation; however, existing methods often result in poor image quality and long inference time. We introduce RepLDM, a novel reprogramming framework for pretrained LDMs that enables high-quality, high-efficiency, high-resolution image generation; see Fig. 1. RepLDM consists of two stages: (i) an attention guidance stage, which generates a latent representation of a higher-quality training-resolution image using a novel training-free self-attention mechanism to enhance the structural consistency; and (ii) a progressive upsampling stage, which progressively performs upsampling in pixel space to mitigate the severe artifacts caused by latent space upsampling. The effective initialization from the first stage allows for denoising at higher resolutions with significantly fewer steps, improving the efficiency. Extensive experimental results demonstrate that RepLDM significantly outperforms state-of-the-art methods in both quality and efficiency for HR image generation, underscoring its advantages for real-world applications. Codes: https://github.com/kmittle/RepLDM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。