arXiv:2605.20659cs.CVcs.LG2026-05被引 1

提出RoPeSLR框架,让扩散模型高效生成超长视频。

RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers

论文配图:RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
图 1 · 摘自论文原文
  • 用3D位置编码引导稀疏低秩注意力,突破传统方法瓶颈。
  • 在90%稀疏度下减少10倍计算量,超长序列推理快2.26倍。
  • 适合需要长视频生成的高效率场景,如大规模视频合成。

扩散Transformer(DiTs)推动了高质量视频生成的发展,但其$\mathcal{O}(L^2)$的注意力复杂度成为长序列生成的主要瓶颈。现有稀疏线性注意力混合方法在极端稀疏下性能显著下降,根源在于标准线性注意力无法保留3D旋转位置编码(RoPE)的正交相对位置结构,削弱了距离感知能力。为此,本文提出3D RoPE驱动的稀疏低秩注意力框架RoPeSLR。在实证假设下,我们发现DiT注意力流形可解耦为一个高频语义尖峰集(稀疏度限制在$\mathcal{O}(L^{3/2})$)与一个极低秩($\mathcal{O}(d_h \log L)$)背景连续体。基于此结构先验,RoPeSLR摒弃标准线性注意力,采用头级低秩参数化并注入可学习的3D绝对位置嵌入,自然建模长程相对距离衰减。该设计保证亚二次稀疏度和亚线性秩增长,特别适合超长视频推理。大量实验验证其可扩展优势:在90%稀疏度下,Wan2.1-1.3B模型减少10倍FLOPs;在HunyuanVideo-13B的超长100K+ token序列上实现2.26倍端到端推理加速,同时保持近乎无损生成质量(平均VBench退化小于1.3%)。

原文摘要 · Abstract (English)

Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, yet their $\mathcal{O}(L^2)$ attention complexity poses a formidable bottleneck for long-sequence synthesis. While recent sparse-linear attention hybrids aim to mitigate this, their performance severely degrades at extreme sparsity due to the "RoPE Dilemma": standard linear attention fails to preserve the orthogonal relative-position structure of 3D Rotary Position Embeddings (RoPE), neutralizing vital distance awareness. To address this, we propose \textbf{RoPeSLR}, a 3D RoPE-guided Sparse-LowRank attention framework. We establish that under empirically validated assumptions, the DiT attention manifold admits a decoupling into a high-frequency semantic spike set (bounded by $\mathcal{O}(L^{3/2})$ sparsity) and an extreme low-rank ($\mathcal{O}(d_h \log L)$) background continuum. Guided by this structural prior, RoPeSLR eschews standard linear attention for a head-wise low-rank parameterization equipped with a learnable 3D Absolute Positional Embedding (PE) injection, seamlessly synthesizing long-range relative distance decay. By guaranteeing sub-quadratic sparsity and sub-linear rank growth, RoPeSLR is exceptionally suited for scaling to ultra-long video inference. Extensive evaluations validate this scalable superiority: at 90\% sparsity, RoPeSLR achieves up to $10\times$ fewer FLOPs on Wan2.1-1.3B and delivers a $2.26\times$ end-to-end inference speedup on the ultra-long 100K+ token sequences of HunyuanVideo-13B, all while maintaining near-lossless generation fidelity (less than 1.3\% average VBench degradation).

扩散模型视频生成注意力机制高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。