提出轻量强制注意力机制,加速自回归视频生成。
Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention
- 设计分块感知增长机制,动态分配注意力稀疏度。
- 在VBench上达到84.5分,端到端速度提升1.2~1.3倍。
- 适合追求高效视频生成的开发者与研究者。
先进的自回归(AR)视频生成模型虽提升了视觉质量和交互性,但注意力机制的二次复杂度仍是部署效率的主要瓶颈。现有稀疏注意力方案在双向模型中表现良好,但在应用于AR模型时导致性能显著下降,原因在于忽视了分块生成的独立性以及对早期有用上下文利用不足。为此,我们提出首个专为AR视频生成设计的稀疏注意力方法——Light Forcing。其引入分块感知增长机制,量化评估每个分块的贡献,动态分配稀疏度;通过渐进式稀疏度增加策略,使当前分块可继承先前分块的知识。此外,提出分层稀疏注意力,以粗到细方式捕捉历史和局部上下文。该两级掩码选择策略(帧级与块级)可自适应处理多样注意力模式。大量实验表明,该方法在质量(如VBench达84.5分)和效率(端到端提速1.2~1.3倍)上均优于现有方案。结合其他高效技术后,进一步实现2.0~3.0倍端到端加速,如在RTX 5090上达27.4 FPS,H100上达33.9 FPS。代码已开源。
原文摘要 · Abstract (English)
Advanced autoregressive (AR) video generation models have improved visual fidelity and interactivity, but the quadratic complexity of attention remains a primary bottleneck for efficient deployment. While existing sparse attention solutions have shown promise on bidirectional models, we identify that applying these solutions to AR models leads to considerable performance degradation for two reasons: isolated consideration of chunk generation and insufficient utilization of past informative context. Motivated by these observations, we propose \textsc{Light Forcing}, the \textit{first} sparse attention solution tailored for AR video generation models. It incorporates a \textit{Chunk-Aware Growth} mechanism to quantitatively estimate the contribution of each chunk, which determines their sparsity allocation. This progressive sparsity increase strategy enables the current chunk to inherit prior knowledge in earlier chunks during generation. Additionally, we introduce a \textit{Hierarchical Sparse Attention} to capture informative historical and local context in a coarse-to-fine manner. Such two-level mask selection strategy (i.e., frame and block level) can adaptively handle diverse attention patterns. Extensive experiments demonstrate that our method outperforms existing sparse attention in quality (e.g., 84.5 on VBench) and efficiency (e.g., $1.2{\sim}1.3\times$ end-to-end speedup). Combined with other efficient solutions, \textsc{Light Forcing} further achieves a $2.0{\sim}3.0\times$ end-to-end speedup across diverse GPUs (e.g., 27.4\,FPS on RTX 5090 and 33.9\,FPS on H100). Code is released via this \href{https://github.com/chengtao-lv/LightForcing}{link}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。