arXiv:2609.06712cs.CV2026-09

提出RoLA,让扩散模型高效处理长视频序列。

RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

论文配图:RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers
图 1 · 摘自论文原文
  • 在旋转位置编码外做低秩线性注意力,保持相对位置关系
  • 90%稀疏度下生成质量不降,推理速度提升2.63倍
  • 无需额外位置参数,适合大规模视频生成任务

扩散Transformer(DiTs)虽能生成高质量视频,但其密集时空自注意力随序列长度呈平方级增长,成为主要推理瓶颈。稀疏低秩混合方法通过局部稀疏分支与全局压缩分支缓解开销。然而,在使用3D旋转位置编码(RoPE)的视频DiTs中,全局分支面临结构兼容性问题:当RoPE应用于非线性特征映射前时,旋转与非线性通常不可交换,难以在保持查询无关线性摘要的同时保留相对旋转几何结构。现有方法常以坐标条件代理或可学习绝对位置模块替代真实跨标记全局聚合,虽有效但近似相对衰减且引入额外位置参数。本文提出RoLA,一种旋转定位的低秩线性注意力分支,既保留真实跨标记聚合,又兼容可复用的线性摘要。该设计将RoPE置于非线性低秩特征映射之外,并复用预训练旋转调度中匹配低秩瓶颈的截断子集。由此实现线性时间的低秩全局分支,天然具备相对位置行为且无额外位置参数;完整稀疏-低秩模块仍包含固定稀疏度的稀疏分支。在开源视频DiTs上的实验表明,该方法在90%稀疏度下仍保持竞争力生成质量,于Wan2.1-14B(720p,81帧)上实现2.63×端到端推理加速(基于NVIDIA H100 GPU测量)。

原文摘要 · Abstract (English)

Diffusion Transformers (DiTs) achieve strong video generation quality, but their dense spatiotemporal self-attention scales quadratically with sequence length and quickly becomes the dominant inference bottleneck. Sparse low-rank hybrids alleviate this cost by combining a local sparse branch with a global compressed branch. In video DiTs equipped with 3D Rotary Position Embeddings (RoPE), the global branch faces a structural compatibility issue: when RoPE is applied before a nonlinear feature map, the rotation and nonlinearity generally do not commute, making it difficult to keep a query-independent linear summary while preserving relative rotary geometry. Existing work often sidesteps this issue by replacing genuine cross-token global aggregation with coordinate-conditioned surrogates or learnable absolute positional modules. These compromises can be effective, but they approximate relative decay from absolute coordinates and introduce extra positional parameters. We propose \textbf{RoLA}, a rotary-positioned low-rank linear-attention branch that keeps genuine cross-token aggregation while remaining compatible with a reusable linear summary. The design applies RoPE \emph{outside} the nonlinear low-rank feature map and reuses a truncated subset of the pre-trained rotary schedule matched to the low-rank bottleneck. This yields a linear-time low-rank global branch with relative positional behavior by design and no additional positional parameters; the full sparse--low-rank module still includes the fixed-sparsity sparse branch. Experiments on open-source video DiTs show that the resulting method remains competitive in generation quality at 90\% sparsity while achieving 2.63$\times$ end-to-end inference speedup on Wan2.1-14B (720p, 81 frames, measured on an NVIDIA H100 GPU).

扩散模型视频生成注意力机制高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。