通过结构化压缩冗余特征，实现扩散模型高效生成
Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration
- 基于结构-细节先验动态合并视觉令牌，减少冗余
- 实现1.55倍加速，图像质量几乎无损
- 无需修改架构，可通用适配各类扩散模型
扩散变换器在视觉生成中表现卓越，但计算成本高昂。现有令牌压缩方法忽略扩散模型的去噪先验，导致加速效果不佳且图像质量下降。本文提出新思路：关注扩散过程未关注区域的特征冗余。基于结构-细节去噪先验分析冗余位置与程度，提出SDTM方法，实现动态视觉令牌合并、压缩比自适应调整和提示重加权。该方法以后训练方式集成，可无缝适配任意DiT架构。跨多种骨干网络、调度器和数据集的实验表明其优越性，例如在保持高质量前提下实现1.55倍加速。
原文摘要 · Abstract (English)
Diffusion transformers have shown exceptional performance in visual generation but incur high computational costs. Token reduction techniques that compress models by sharing the denoising process among similar tokens have been introduced. However, existing approaches neglect the denoising priors of the diffusion models, leading to suboptimal acceleration and diminished image quality. This study proposes a novel concept: attend to prune feature redundancies in areas not attended by the diffusion process. We analyze the location and degree of feature redundancies based on the structure-then-detail denoising priors. Subsequently, we introduce SDTM, a structure-then-detail token merging approach that dynamically compresses feature redundancies. Specifically, we design dynamic visual token merging, compression ratio adjusting, and prompt reweighting for different stages. Served in a post-training way, the proposed method can be integrated seamlessly into any DiT architecture. Extensive experiments across various backbones, schedulers, and datasets showcase the superiority of our method, for example, it achieves 1.55 times acceleration with negligible impact on image quality. Project page: https://github.com/ICTMCG/SDTM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。