通过自我反思式采样,扩散模型可自提升生成质量与提示对齐度。
Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection
- 交替执行去噪与反演,利用两者差异捕捉提示语义信息。
- 新方法Z-Sampling在多个数据集上提升生成质量,最高使胜率达94%。
- 无需修改模型即可通用接入,计算开销极低,适合各类扩散模型。
扩散模型作为当前主流生成范式,可通过条件信息引导潜在空间向目标方向演化。然而,现有文本到图像的扩散模型在处理复杂提示时,常难以兼顾图像质量和提示对齐。为此,本文提出扩散自我反思机制,通过交替进行去噪与反演操作,理论与实证证明其可利用去噪与反演之间的指导差距,有效捕获与提示相关的语义信息。基于此,我们推导出一种新型自反思采样方法——Zigzag Diffusion Sampling(Z-Sampling),该方法沿采样路径逐步累积语义信息,显著提升生成效果。作为即插即用方案,Z-Sampling可广泛适配多种扩散模型(如加速型、Transformer-based模型),编码与计算成本极低。大量实验表明,其能在多个基准数据集、模型及评估指标上普遍且显著提升生成质量。例如,使用Z-Sampling的DreamShaper在HPSv2评测中胜率高达94%,超越原始结果;同时,其还可与Diffusion-DPO等其他正交方法协同增强性能。
原文摘要 · Abstract (English)
Diffusion models, the most popular generative paradigm so far, can inject conditional information into the generation path to guide the latent towards desired directions. However, existing text-to-image diffusion models often fail to maintain high image quality and high prompt-image alignment for those challenging prompts. To mitigate this issue and enhance existing pretrained diffusion models, we mainly made three contributions in this paper. First, we propose diffusion self-reflection that alternately performs denoising and inversion and demonstrate that such diffusion self-reflection can leverage the guidance gap between denoising and inversion to capture prompt-related semantic information with theoretical and empirical evidence. Second, motivated by theoretical analysis, we derive Zigzag Diffusion Sampling (Z-Sampling), a novel self-reflection-based diffusion sampling method that leverages the guidance gap between denosing and inversion to accumulate semantic information step by step along the sampling path, leading to improved sampling results. Moreover, as a plug-and-play method, Z-Sampling can be generally applied to various diffusion models (e.g., accelerated ones and Transformer-based ones) with very limited coding and computational costs. Third, our extensive experiments demonstrate that Z-Sampling can generally and significantly enhance generation quality across various benchmark datasets, diffusion models, and performance evaluation metrics. For example, DreamShaper with Z-Sampling can self-improve with the HPSv2 winning rate up to 94% over the original results. Moreover, Z-Sampling can further enhance existing diffusion models combined with other orthogonal methods, including Diffusion-DPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。