通过分阶段渐进式采样,高效生成高分辨率图像。
FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling
- 分阶段逐步提升分辨率,逐级扩展频带并细化细节。
- 生成2048×2048图像速度比ScaleCrafter快2.86倍,FID_b提升11.6。
- 适用于SDXL、SD3等复杂模型,适合追求高效高质生成的用户。
尽管扩散模型在图像生成上取得巨大成功,但生成高于训练分辨率的图像仍面临计算成本高的挑战。现有方法通常在整个采样过程中保持全分辨率,并同时处理所有频率成分,违背了潜在扩散模型固有的粗到细特性,在早期扩散阶段浪费计算资源处理过早的高频细节。为此,我们提出一种高效的频率感知级联采样框架FreCaS,用于更高分辨率的图像生成。FreCaS将采样过程分解为多个分辨率逐步提升的级联阶段,逐级扩展频率带并精炼对应细节。我们提出一种创新的频率感知无分类器引导(FA-CFG)策略,为不同频率成分分配不同的引导强度,指导扩散模型在每个阶段的扩展频率域中添加新细节。此外,融合前一阶段与当前阶段的交叉注意力图,避免生成不忠实的布局。实验表明,FreCaS在图像质量和生成速度上均显著优于现有最优方法。尤其在使用预训练SDXL模型生成2048×2048图像时,FreCaS分别比ScaleCrafter和DemoFusion快2.86倍和6.07倍,FID_b分别提升11.6和3.7。FreCaS可轻松扩展至更复杂的模型如SD3。代码已公开于https://github.com/xtudbxk/FreCaS。
原文摘要 · Abstract (English)
While image generation with diffusion models has achieved a great success, generating images of higher resolution than the training size remains a challenging task due to the high computational cost. Current methods typically perform the entire sampling process at full resolution and process all frequency components simultaneously, contradicting with the inherent coarse-to-fine nature of latent diffusion models and wasting computations on processing premature high-frequency details at early diffusion stages. To address this issue, we introduce an efficient $\textbf{Fre}$quency-aware $\textbf{Ca}$scaded $\textbf{S}$ampling framework, $\textbf{FreCaS}$ in short, for higher-resolution image generation. FreCaS decomposes the sampling process into cascaded stages with gradually increased resolutions, progressively expanding frequency bands and refining the corresponding details. We propose an innovative frequency-aware classifier-free guidance (FA-CFG) strategy to assign different guidance strengths for different frequency components, directing the diffusion model to add new details in the expanded frequency domain of each stage. Additionally, we fuse the cross-attention maps of previous and current stages to avoid synthesizing unfaithful layouts. Experiments demonstrate that FreCaS significantly outperforms state-of-the-art methods in image quality and generation speed. In particular, FreCaS is about 2.86$\times$ and 6.07$\times$ faster than ScaleCrafter and DemoFusion in generating a 2048$\times$2048 image using a pre-trained SDXL model and achieves an FID$_b$ improvement of 11.6 and 3.7, respectively. FreCaS can be easily extended to more complex models such as SD3. The source code of FreCaS can be found at https://github.com/xtudbxk/FreCaS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。