提出新方法DiTo,让扩散模型生成更清晰图像的同时提速
Rethinking Token Reduction for Diffusion Models via Output-Similarity-Awareness

- 基于输出相似性设计跨时间步的令牌匹配策略
- 在相同加速比下,图像质量提升1.6至3.9分贝
- 适合追求高效高质图像生成的研究者与应用开发
扩散变压器(DiTs)虽能生成高质量图像,但其计算复杂度随令牌数呈二次增长。现有令牌缩减(TR)方法仅依赖输入相似性,忽视生成目标——最小化恢复误差,导致与输出特性脱节。为此,我们提出DiTo,一种以输出为中心的新型缩减范式。观察发现相邻时间步间输出令牌相似性保持稳定,因此利用前一步相似性作为匹配时刻的代理,建立对应关系并复用于多个后续缩减时刻。为优化交错调度,提出基于配对匹配率(PMR)的间隔调度策略以确定最优匹配频率。此外,为缓解重复使用带来的局部近似误差和块状伪影,引入频率感知令牌匹配,加入选择频率惩罚项。大量实验表明,DiTo在相同加速比下始终优于现有方法,峰值信噪比(PSNR)提升1.6–3.9 dB,达成更优帕累托前沿。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) achieve superior image generation quality but suffer from quadratic computational complexity relative to token count. While various token reduction (TR) methods have been proposed to mitigate this cost, they overlook the primary objective of generative models: minimizing recovery error, which requires reflecting output token similarity. They rely solely on input token similarity inherited from reduction-only ViT paradigms, leading to a fundamental misalignment with this objective. To bridge this gap, we propose DiTo, a novel TR paradigm that shifts the focus toward output-centric token reduction. Based on the observation that output token similarity is consistently preserved across adjacent timesteps, DiTo utilizes prior-step similarities as an effective proxy to establish token correspondences at a Matching timestep, which are then reused across multiple subsequent Reduction timesteps. To optimize this interleaved scheduling, we propose Pair Match Ratio (PMR)-guided Interval Scheduling to determine the optimal matching frequency. Furthermore, to mitigate localized approximation errors and resulting blocking artifacts caused by repeated reuse, we propose Frequency-aware Token Matching by incorporating a selection-frequency penalty. Extensive experiments demonstrate that DiTo consistently outperforms existing TR methods with 1.6-3.9 dB higher PSNR at comparable speedups, achieving a superior Pareto frontier.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。