无需训练的动态稀疏注意力,加速视频生成推理
Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification

- 在线软最大化中动态阈值选择关键注意力块
- 视频生成和编辑分别提速2.1倍和2.3倍
- 无需预计算,适合部署在资源受限场景
扩散变换器是高质量视频生成的核心,但长序列导致注意力成为主要推理瓶颈。现有无训练动态稀疏注意力方法在效率与精度间难以兼顾:(1)路由机制僵化或成本高,固定比例选取或按累积概率保留导致预算不均衡且需大量计算代理分数;(2)丢弃未选块造成精度损失。本文提出Sol-Attn,在一次在线软最大化中统一动态路由、稀疏计算与近似校正,通过代理分数复用实现无材料化动态预算控制,并重用未选块代理分数近似其贡献。在图像与视频生成任务上验证,相比现有方法显著提升质量-效率平衡,视频生成与编辑分别实现2.1倍和2.3倍端到端加速,同时保持视觉质量。
原文摘要 · Abstract (English)
Diffusion transformers are essential for high-fidelity video generation, but long token sequences make attention a dominant inference bottleneck. Training-free dynamic sparse attention alleviates this bottleneck by computing only selected key-value blocks, yet existing methods struggle to sparsify attention both efficiently and accurately for two reasons: (1) Rigid, unpredictable, and costly routing: selecting a fixed fraction of top-ranked blocks by proxy score imposes fixed budgets, whereas retaining blocks to reach a target cumulative proxy probability mass yields dynamic but potentially imbalanced budgets; both incur non-negligible overhead from computing and materializing proxy scores. (2) Lossy keep-or-drop sparsification: unselected blocks are discarded entirely, degrading accuracy under aggressive sparsity. These limitations motivate cheaper dynamic-budget routing while limiting accuracy degradation. In this paper, we introduce training-free Sol-Attn (Sparsifying online attention), which unifies dynamic routing, sparse computation, and approximation correction in a single online-softmax pass, achieving a better accuracy-efficiency trade-off in sparse attention. The core of Sol-Attn is on-the-fly block thresholding with proxy-score reuse, which selects critical blocks by comparing block proxy scores against a threshold during online softmax. This design enables dynamic yet controllable block budgets without materializing the proxy map, while directly reusing the proxy scores of unselected blocks to approximate their contribution. Experiments across image and video generation tasks show that Sol-Attn advances the quality-efficiency frontier of training-free sparse attention, delivering 2.1 times and 2.3 times end-to-end speedups for video generation and editing, respectively, while preserving visual quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。