提出自适应稀疏注意力系统，显著提升视频生成多卡推理效率。
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
- 采用动态Top-p路由+安全底限，结合视频块结构实现稀疏路由。
- 运行时负载均衡将重头任务通过点对点通信迁移，减少关键路径延迟。
- 适合大规模视频生成模型的多卡部署，尤其优化长序列推理性能。
视频扩散变换器处理长时空序列，自注意力成为高分辨率视频生成的主要瓶颈。无训练的稀疏注意力虽降低计算开销，但在多卡序列并行下，自适应Top-p路由导致各注意力头负载不均，引发秩级延迟问题。本文提出 extit{FVAttn}，一种无需训练的稀疏注意力系统，可显著提升多卡环境下自适应稀疏注意力的分布式执行效率。该方法在前端使用Top-p路由、Top-k安全底限与视频感知块组织，在运行时修复已生成掩码；通过运行时负载均衡，将少数重头任务经点对点通信迁移以缩短关键路径；同时利用空闲资源填充非关键秩的剩余空隙，并通过重叠调度与迁移开销来隐藏延迟。在经过步骤蒸馏的Wan2.2 I2V模型上， extit{FVAttn}将平均负载不均衡从1.34降至1.08，相比FlashAttention实现4.41倍注意力加速，同时带来2.02–2.11倍的DiT推理加速，且视频质量保持竞争力。
原文摘要 · Abstract (English)
Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-$p$ routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present \method{}, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. \method{} uses Top-$p$ routing, a Top-$k$ safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, \method{} reduces average load imbalance from 1.34 to 1.08 and delivers a $4.41\times$ attention speedup over FlashAttention, while achieving a $2.02$--$2.11\times$ DiT inference speedup with competitive video quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。