通过动态合并相似令牌，实现扩散模型加速且不丢质量。
Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Model
- 根据相似度自适应选择合并令牌，提升效率。
- 在去噪过程提速24%，FID分数不变。
- 无需训练，适合部署于现有扩散模型。
扩散模型在生成高质量高维图像方面展现出巨大潜力，但其高昂的计算成本和缓慢的推理速度限制了应用，主要源于自注意力机制随输入规模呈二次增长的复杂度。为缓解此问题，已有多种方法被提出，其中一种是减少输入自注意力的令牌数量，称为令牌合并（ToMe）。本文提出缓存自适应令牌合并（CA-ToMe），通过计算令牌间的相似度，合并最相似的r比例令牌。针对相邻步骤中重复模式及相似度频率变化的问题，我们引入自适应合并阈值与跨多步缓存相似对的机制。实验表明，本方法为无需训练的加速技术，在去噪过程中实现1.24倍的速度提升，同时保持与现有方法相当的FID分数。
原文摘要 · Abstract (English)
Diffusion models have emerged as a promising approach for generating high-quality, high-dimensional images. Nevertheless, these models are hindered by their high computational cost and slow inference, partly due to the quadratic computational complexity of the self-attention mechanisms with respect to input size. Various approaches have been proposed to address this drawback. One such approach focuses on reducing the number of tokens fed into the self-attention, known as token merging (ToMe). In our method, which is called cached adaptive token merging(CA-ToMe), we calculate the similarity between tokens and then merge the r proportion of the most similar tokens. However, due to the repetitive patterns observed in adjacent steps and the variation in the frequency of similarities, we aim to enhance this approach by implementing an adaptive threshold for merging tokens and adding a caching mechanism that stores similar pairs across several adjacent steps. Empirical results demonstrate that our method operates as a training-free acceleration method, achieving a speedup factor of 1.24 in the denoising process while maintaining the same FID scores compared to existing approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。