通过挖掘视频隐空间相关性，实现视频生成模型的软硬件协同加速。
Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations
- 利用通道级时空相关性，设计轻量重用算法跳过冗余计算。
- 在保持生成质量高于17 dB的前提下，实现最高5.9倍加速和16倍节能。
- 适合关注视频生成推理效率与能效的开发者与芯片设计者。
视频扩散变换器（vDiTs）生成高质量视频，但因长扩散步数和自注意力计算导致极高计算开销。当扩散步数减少后，自注意力计算成为主要瓶颈。现有加速方法多沿用大语言模型的稀疏注意力技术，未考虑视频数据独特的时空相关性。本文提出Kaleido，一种算法-硬件协同设计，通过挖掘隐空间中的通道级时空相关性，加速vDiTs中所有操作。基于此，我们提出一种轻量级通道重用算法，通过复用部分结果跳过冗余计算，同时生成质量优于之前方法（>17 dB）。为高效支持该算法，设计了可重构处理单元的类阵列加速器及轻量数据调度器，以缓解重用算法带来的不规则稀疏性和数据访问模式。在三个主流vDiT模型上的评估显示，Kaleido相比现有最优加速器实现最高5.9倍加速和16.0倍能效提升。
原文摘要 · Abstract (English)
Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost due to the long diffusion timesteps and self attention computation. As diffusion timesteps are reduced, the computation cost of self attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data. This paper presents Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channelwise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。