提出VORTA框架，让视频扩散模型更快生成，不降质还兼容其他加速方法。
VORTA: Efficient Video Diffusion via Routing Sparse Attention
- 用稀疏注意力捕捉长距离依赖，减少计算开销。
- 在VBench上实现1.76倍加速，结合其他方法最高达14.41倍。
- 适合需要高效视频生成的开发者和研究者使用。
视频扩散变换器在高质量视频生成方面取得显著进展，但因高维视频序列上注意力机制的二次复杂度，仍存在计算成本高昂的问题。现有加速方法虽利用注意力得分的局部稀疏性提升效率，却难以有效加速长程计算。为此，我们提出VORTA，包含两项新组件：1）高效捕捉长距离依赖的稀疏注意力机制；2）自适应将全3D注意力替换为特定稀疏注意力变体的路由策略。VORTA在VBench上实现端到端 1.76 倍加速，且无质量损失。同时可无缝集成多种其他加速技术（如模型缓存、步数蒸馏），最高可达 14.41 倍加速，性能下降可忽略。VORTA显著提升视频扩散变换器在实际场景中的实用性。代码与权重已公开于 https://github.com/wenhao728/VORTA。
原文摘要 · Abstract (English)
Video diffusion transformers have achieved remarkable progress in high-quality video generation, but remain computationally expensive due to the quadratic complexity of attention over high-dimensional video sequences. Recent acceleration methods enhance the efficiency by exploiting the local sparsity of attention scores; yet they often struggle with accelerating the long-range computation. To address this problem, we propose VORTA, an acceleration framework with two novel components: 1) a sparse attention mechanism that efficiently captures long-range dependencies, and 2) a routing strategy that adaptively replaces full 3D attention with specialized sparse attention variants. VORTA achieves an end-to-end speedup $1.76\times$ without loss of quality on VBench. Furthermore, it can seamlessly integrate with various other acceleration methods, such as model caching and step distillation, reaching up to speedup $14.41\times$ with negligible performance degradation. VORTA demonstrates its efficiency and enhances the practicality of video diffusion transformers in real-world settings. Codes and weights are available at https://github.com/wenhao728/VORTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。