arXiv:2604.08048cs.CV2026-04中稿 · CVPR被引 2

通过交换令牌实现无需条件的图像生成优化,提升画质与控制精度。

Guiding a Diffusion Model by Swapping Its Tokens

  • 通过交换语义差异大的令牌潜变量生成扰动预测
  • 在多个数据集上优于现有无条件方法,提升图像质量和提示对齐度
  • 可插件式接入任意扩散模型,适合需要精细控制的生成任务

Classifier-Free Guidance(CFG)是一种广泛使用的推理阶段技术,用于提升扩散模型的图像质量。然而,其依赖文本条件的特性限制了其在无条件生成中的应用。本文提出一种简单方法,使CFG式的引导适用于有条件和无条件生成。核心思想是通过简单的令牌交换操作生成扰动预测,并利用其与原始预测之间的方向,引导采样趋向更高保真度的分布。具体而言,我们在空间或通道维度上交换最语义相异的令牌潜变量对。与现有全局或无约束扰动方法不同,本方法选择性地交换并重组令牌,实现更精细的扰动控制及其对生成样本的影响。在MS-COCO 2014、MS-COCO 2017和ImageNet数据集上的实验表明,所提出的Self-Swap Guidance(SSG)在主流扩散模型上均优于先前的无条件方法,在不同设置下提升了图像保真度与提示对齐度。其细粒度扰动机制还增强了鲁棒性,在更广范围的扰动强度下减少副作用。总体而言,SSG将CFG扩展至更广泛的应用场景,支持有/无条件生成,且可作为即插即用模块,为任意扩散模型带来即时性能提升。

原文摘要 · Abstract (English)

Classifier-Free Guidance (CFG) is a widely used inference-time technique to boost the image quality of diffusion models. Yet, its reliance on text conditions prevents its use in unconditional generation. We propose a simple method to enable CFG-like guidance for both conditional and unconditional generation. The key idea is to generate a perturbed prediction via simple token swap operations, and use the direction between it and the clean prediction to steer sampling towards higher-fidelity distributions. In practice, we swap pairs of most semantically dissimilar token latents in either spatial or channel dimensions. Unlike existing methods that apply perturbation in a global or less constrained manner, our approach selectively exchanges and recomposes token latents, allowing finer control over perturbation and its influence on generated samples. Experiments on MS-COCO 2014, MS-COCO 2017, and ImageNet datasets demonstrate that the proposed Self-Swap Guidance (SSG), when applied to popular diffusion models, outperforms previous condition-free methods in image fidelity and prompt alignment under different set-ups. Its fine-grained perturbation granularity also improves robustness, reducing side-effects across a wider range of perturbation strengths. Overall, SSG extends CFG to a broader scope of applications including both conditional and unconditional generation, and can be readily inserted into any diffusion model as a plug-in to gain immediate improvements.

扩散模型图像生成无条件生成引导机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。