动态调整扩散模型加速策略,兼顾速度与生成质量。
SODA: Sensitivity-Oriented Dynamic Acceleration for Diffusion Transformer
- 基于细粒度敏感性建模,自适应决定缓存与剪枝时机
- 在多个模型上实现提速同时保持最高生成质量
- 适合需要高效高质图像生成的开发者和研究者
扩散Transformer已成为视觉生成的主流范式,但其推理效率低下仍是关键瓶颈。现有训练无关技术中,缓存虽加速快却常牺牲质量,剪枝则相反。将二者结合可平衡加速与质量,但现有方法多采用固定启发式策略,难以捕捉模型在时间步、层和模块上的精细敏感性变化,导致关键计算被忽略,质量下降,且泛化能力差。为此,本文提出SODA:一种面向敏感性的动态加速方法,通过离线构建跨时间步、层和模块的敏感性误差模型,以敏感性误差为代价函数,用动态规划优化缓存间隔,最小化缓存对敏感性的影响。在剪枝与缓存重用时,自适应确定剪枝时机与速率,保留高敏感性令牌的计算,显著提升生成保真度。在DiT-XL/2、PixArt-α和OpenSora上大量实验表明,SODA在可控加速比下达到当前最优生成质量。代码已公开于https://github.com/leaves162/SODA。
原文摘要 · Abstract (English)
Diffusion Transformers have become a dominant paradigm in visual generation, yet their low inference efficiency remains a key bottleneck hindering further advancement. Among common training-free techniques, caching offers high acceleration efficiency but often compromises fidelity, whereas pruning shows the opposite trade-off. Integrating caching with pruning achieves a balance between acceleration and generation quality. However, existing methods typically employ fixed and heuristic schemes to configure caching and pruning strategies. While they roughly follow the overall sensitivity trend of generation models to acceleration, they fail to capture fine-grained and complex variations, inevitably skipping highly sensitive computations and leading to quality degradation. Furthermore, such manually designed strategies exhibit poor generalization. To address these issues, we propose SODA, a Sensitivity-Oriented Dynamic Acceleration method that adaptively performs caching and pruning based on fine-grained sensitivity. SODA builds an offline sensitivity error modeling framework across timesteps, layers, and modules to capture the sensitivity to different acceleration operations. The cache intervals are optimized via dynamic programming with sensitivity error as the cost function, minimizing the impact of caching on model sensitivity. During pruning and cache reuse, SODA adaptively determines the pruning timing and rate to preserve computations of highly sensitive tokens, significantly enhancing generation fidelity. Extensive experiments on DiT-XL/2, PixArt-$α$, and OpenSora demonstrate that SODA achieves state-of-the-art generation fidelity under controllable acceleration ratios. Our code is released publicly at: https://github.com/leaves162/SODA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。