发现扩散模型中各步激活值冗余差异大,提出针对性分析工具。
Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study
- 分析不同扩散模型在各生成步骤的激活冗余分布特征。
- 发现同一模型冗余模式稳定,不受提示词、步数和调度影响。
- 开发可定制缓存策略的分析工具,适配多种模型架构。
扩散变换器(DiTs)因模型容量增大和生成高分辨率图像/视频的需求,导致推理延迟显著上升,影响实时性能。已有研究指出相邻扩散步骤间激活值存在高度相似性(称为冗余),并提出多种缓存机制以降低计算开销,但现有文献对冗余的探索有限,且结果难以跨模型通用。本研究针对此问题,对主流DiT模型进行了系统性冗余分析。实验表明,不同模型间冗余分布差异显著;而在单个模型内部,冗余分布对输入提示词、步数或调度策略变化保持稳定。由于缺乏跨模型一致性模式,针对特定模型群设计的缓存策略难以迁移。为此,我们提出一个用于分析单个模型冗余特性的工具,支持后续研究为具体架构定制高效缓存方案。项目已开源:https://github.com/xdit-project/DiTCacheAnalysis。
原文摘要 · Abstract (English)
The increased model capacity of Diffusion Transformers (DiTs) and the demand for generating higher resolutions of images and videos have led to a significant rise in inference latency, impacting real-time performance adversely. While prior research has highlighted the presence of high similarity in activation values between adjacent diffusion steps (referred to as redundancy) and proposed various caching mechanisms to mitigate computational overhead, the exploration of redundancy in existing literature remains limited, with findings often not generalizable across different DiT models. This study aims to address this gap by conducting a comprehensive investigation into redundancy across a broad spectrum of mainstream DiT models. Our experimental analysis reveals substantial variations in the distribution of redundancy across diffusion steps among different DiT models. Interestingly, within a single model, the redundancy distribution remains stable regardless of variations in input prompts, step counts, or scheduling strategies. Given the lack of a consistent pattern across diverse models, caching strategies designed for a specific group of models may not easily transfer to others. To overcome this challenge, we introduce a tool for analyzing the redundancy of individual models, enabling subsequent research to develop tailored caching strategies for specific model architectures. The project is publicly available at https://github.com/xdit-project/DiTCacheAnalysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。