不训练即可提升扩散模型高分辨率生成准确性,解决重复与局部失真问题。
AccDiffusion v2: Towards More Accurate Higher-Resolution Diffusion Extrapolation
- 将全局提示拆解为局部块级提示,提升生成细节精度。
- 引入ControlNet辅助结构信息,减轻局部失真现象。
- 采用带窗口交互的稀疏采样,增强全局语义一致性,适合高分辨率图像生成场景。
扩散模型在推理分辨率与预训练分辨率不一致时,会出现严重的物体重复和局部失真问题。本文提出AccDiffusion v2,一种无需训练的逐块高分辨率扩散外推方法。深入分析表明,使用相同文本提示生成不同块会导致重复生成,而无提示则削弱图像细节。为此,AccDiffusion v2首次将原始图像内容感知提示解耦为一组块内容感知提示,更精准描述每个局部区域。进一步分析发现,局部失真源于对高分辨率图像局部结构描述不准确。为此,AccDiffusion v2首次在高分辨率扩散外推中引入ControlNet辅助局部结构信息,以缓解局部失真。最后,分析表明全局语义信息有助于抑制重复与失真,因此提出带有窗口交互的稀疏采样策略,以更好获取全局语义。大量实验(定量与定性)证明,AccDiffusion v2在无需训练的情况下达到当前最优性能,有效抑制了重复生成与局部失真问题。代码已开源:https://github.com/lzhxmu/AccDiffusion_v2。
原文摘要 · Abstract (English)
Diffusion models suffer severe object repetition and local distortion when the inference resolution differs from its pre-trained resolution. We propose AccDiffusion v2, an accurate method for patch-wise higher-resolution diffusion extrapolation without training. Our in-depth analysis in this paper shows that using an identical text prompt for different patches leads to repetitive generation, while the absence of a prompt undermines image details. In response, our AccDiffusion v2 novelly decouples the vanilla image-content-aware prompt into a set of patch-content-aware prompts, each of which serves as a more precise description of a patch. Further analysis reveals that local distortion arises from inaccurate descriptions in prompts about the local structure of higher-resolution images. To address this issue, AccDiffusion v2, for the first time, introduces an auxiliary local structural information through ControlNet during higher-resolution diffusion extrapolation aiming to mitigate the local distortions. Finally, our analysis indicates that global semantic information is conducive to suppressing both repetitive generation and local distortion. Hence, our AccDiffusion v2 further proposes dilated sampling with window interaction for better global semantic information during higher-resolution diffusion extrapolation. We conduct extensive experiments, including both quantitative and qualitative comparisons, to demonstrate the efficacy of our AccDiffusion v2. The quantitative comparison shows that AccDiffusion v2 achieves state-of-the-art performance in image generation extrapolation without training. The qualitative comparison intuitively illustrates that AccDiffusion v2 effectively suppresses the issues of repetitive generation and local distortion in image generation extrapolation. Our code is available at https://github.com/lzhxmu/AccDiffusion_v2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。