arXiv:2501.05680cs.ARcs.AI2025-01被引 16

通过挖掘扩散模型的迭代间与迭代内稀疏性,显著提升生成效率与能效。

EXION: Exploiting Inter- and Intra-Iteration Output Sparsity for Diffusion Models

  • 利用迭代间与迭代内输出稀疏性,设计轻量级软件优化算法。
  • 在多模态扩散模型上保持精度不变,性能与能效提升最高达379倍。
  • 适合追求低延迟、高能效的图像/视频生成应用部署。

近年来,扩散模型已成为从文本提示生成多样化多模态输出的新型AI解决方案。尽管能力强大,其迭代架构导致计算开销大,存在延迟高、能耗高的问题。现有Transformer加速方法无法解决扩散模型的迭代本质。本文提出EXION,首个软硬件协同设计的扩散模型加速器,通过挖掘扩散模型中独特的迭代间与迭代内输出稀疏性来解决计算挑战。我们提出两项软件优化:一是FFN-Reuse算法,识别并跳过不同迭代间前馈网络中的冗余计算(迭代间稀疏性);二是改进的急切预测方法,采用两步领先1检测精确预测注意力分数,跳过单次迭代内的无效计算(迭代内稀疏性)。同时引入名为ConMerge的新数据压缩机制,将稀疏矩阵合并为紧凑形式以提升硬件利用率。最终,硬件架构专门支持上述稀疏性诱导算法,实现高输出稀疏性向能效与性能的高效转化。为验证可行性,我们在多种多模态扩散模型上证明了精度无损失。在服务器与边缘设备上实例化后,与同规格GPU对比,EXION在性能上提升3.2–379.3倍,能效提升45.1–3067.6倍(服务器);在边缘端分别提升42.6–1090.9倍与196.9–4668.2倍。

原文摘要 · Abstract (English)

Over the past few years, diffusion models have emerged as novel AI solutions, generating diverse multi-modal outputs from text prompts. Despite their capabilities, they face challenges in computing, such as excessive latency and energy consumption due to their iterative architecture. Although prior works specialized in transformer acceleration can be applied, the iterative nature of diffusion models remains unresolved. In this paper, we present EXION, the first SW-HW co-designed diffusion accelerator that solves the computation challenges by exploiting the unique inter- and intra-iteration output sparsity in diffusion models. To this end, we propose two SW-level optimizations. First, we introduce the FFN-Reuse algorithm that identifies and skips redundant computations in FFN layers across different iterations (inter-iteration sparsity). Second, we use a modified eager prediction method that employs two-step leading-one detection to accurately predict the attention score, skipping unnecessary computations within an iteration (intra-iteration sparsity). We also introduce a novel data compaction mechanism named ConMerge, which can enhance HW utilization by condensing and merging sparse matrices into compact forms. Finally, it has a dedicated HW architecture that supports the above sparsity-inducing algorithms, translating high output sparsity into improved energy efficiency and performance. To verify the feasibility of the EXION, we first demonstrate that it has no impact on accuracy in various types of multi-modal diffusion models. We then instantiate EXION in both server- and edge-level settings and compare its performance against GPUs with similar specifications. Our evaluation shows that EXION achieves dramatic improvements in performance and energy efficiency by 3.2-379.3x and 45.1-3067.6x compared to a server GPU and by 42.6-1090.9x and 196.9-4668.2x compared to an edge GPU.

扩散模型稀疏性能效优化硬件加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。