arXiv:2506.04648cs.CV2025-06NeurIPS被引 18

通过联合优化8位浮点量化与稀疏性,显著加速视频生成模型推理。

FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion

  • 设计3D分块统一粒度,同时支持量化和稀疏化。
  • 根据去噪步骤动态调整,减少误差累积,保持生成质量。
  • 适配Hopper架构的高效内核,实现7倍注意力层加速。

扩散生成模型已成为高质量连贯视频内容的标准,但其推理速度慢、计算开销大,制约实际部署。尽管量化和稀疏化各自可加速推理并保持生成质量,但现有无训练方法直接结合二者会导致性能显著下降,因缺乏联合优化。本文提出FPSAttention,一种面向视频生成的训练感知型FP8量化与稀疏性协同设计,聚焦3D双向注意力机制。核心创新包括:1)统一3D分块粒度,同步支持量化与稀疏;2)去噪步骤感知策略,适应噪声调度,缓解量化/稀疏误差与去噪步之间的强相关性;3)原生硬件友好内核,结合FlashAttention并优化利用Hopper架构特性,实现高效执行。在Wan2.1的1.3B和14B模型上训练,于VBench基准测试中,相比BF16基线,在720p分辨率下实现注意力操作7.09倍内核加速,端到端视频生成4.96倍加速,且未牺牲生成质量。

原文摘要 · Abstract (English)

Diffusion generative models have become the standard for producing high-quality, coherent video content, yet their slow inference speeds and high computational demands hinder practical deployment. Although both quantization and sparsity can independently accelerate inference while maintaining generation quality, naively combining these techniques in existing training-free approaches leads to significant performance degradation due to the lack of joint optimization. We introduce FPSAttention, a novel training-aware co-design of FP8 quantization and sparsity for video generation, with a focus on the 3D bi-directional attention mechanism. Our approach features three key innovations: 1) A unified 3D tile-wise granularity that simultaneously supports both quantization and sparsity; 2) A denoising step-aware strategy that adapts to the noise schedule, addressing the strong correlation between quantization/sparsity errors and denoising steps; 3) A native, hardware-friendly kernel that leverages FlashAttention and is implemented with optimized Hopper architecture features for highly efficient execution. Trained on Wan2.1's 1.3B and 14B models and evaluated on the VBench benchmark, FPSAttention achieves a 7.09x kernel speedup for attention operations and a 4.96x end-to-end speedup for video generation compared to the BF16 baseline at 720p resolution-without sacrificing generation quality.

视频生成扩散模型量化稀疏性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。