arXiv:2502.04507cs.CV2025-02ICML被引 119

提出滑动块注意力机制,让视频生成速度提升近3倍,且不损失画质。

Fast Video Generation with Sliding Tile Attention

  • 用局部滑动块替代全时空间注意力,减少冗余计算。
  • 在720P 5秒视频生成中,推理时间从945秒降至685秒,提速2.8至17倍。
  • 无需重训练即可加速,适合部署高帧率视频生成系统。

具有3D全注意力的扩散变换器(DiTs)虽能实现顶尖视频生成效果,但计算成本极高——仅生成5秒720P视频,注意力计算就占总推理时间945秒中的800秒。本文提出滑动块注意力(STA),利用预训练视频扩散模型中注意力得分集中于局部3D窗口的特性,通过滑动并逐块关注时空局部区域,消除全注意力冗余。与传统逐标记滑动窗口注意力不同,STA采用新型硬件友好型滑动窗口设计,按块处理,在保持表达能力的同时实现高效硬件计算。经精心内核级优化,首次实现高效的2D/3D滑动窗口注意力,达到58.79%的模型浮点利用率(MFU)。STA相比FlashAttention-2(FA2)加速2.8–17倍,相比FlashAttention-3(FA3)加速1.6–10倍。在主流视频DiT模型HunyuanVideo上,端到端延迟由FA3的945秒降至685秒,未降低质量;启用微调后进一步降至268秒,仅在VBench上下降0.09%。代码已开源:https://github.com/hao-ai-lab/FastVideo。

原文摘要 · Abstract (English)

Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art video generation, but suffer from prohibitive compute cost -- when generating just a 5-second 720P video, attention alone takes 800 out of 945 seconds of total inference time. This paper introduces sliding tile attention (STA) to address this challenge. STA leverages the observation that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows. By sliding and attending over the local spatial-temporal region, STA eliminates redundancy from full attention. Unlike traditional token-wise sliding window attention (SWA), STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. With careful kernel-level optimizations, STA offers the first efficient 2D/3D sliding-window-like attention implementation, achieving 58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s without quality degradation, requiring no training. Enabling finetuning further lowers latency to 268s with only a 0.09% drop on VBench. We make our codebase public at https://github.com/hao-ai-lab/FastVideo.

视频生成注意力机制加速推理扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。