arXiv:2605.30351cs.CVcs.AI2026-05被引 1

用低秩潜变量压缩视频扩散的键值缓存,显著降低内存占用。

VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion

论文配图:VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion
图 1 · 摘自论文原文
  • 提出多头潜注意力机制,共享低秩内容潜变量和3D-RoPE位置键
  • 每层缓存减少92.7%键值内存,长时序生成质量接近基线
  • 适合追求高吞吐、低内存消耗的视频生成应用

长时序因果视频扩散普遍采用固定大小滑动窗口键值缓存,近期进展主要在窗口内令牌选择或位置编码方式上创新。而每头键值布局这一关键内存与延迟来源却基本未变。本文首次研究视频扩散中的多头潜注意力(MLA)。VideoMLA将每头的键值替换为共享的低秩内容潜变量与解耦的3D-RoPE位置键,使每令牌缓存内存减少92.7%。我们进一步探究了为何尽管预训练视频注意力不满足低秩假设(99%能量有效秩远超潜空间维度),MLA仍能保持质量:其瓶颈由压缩结构决定,而非预训练谱;无论谱或随机初始化,均从初始化即占据几乎全部秩预算,训练仅在此内适配。在VBench上,VideoMLA匹配短时程基线,在长时程中表现最优,单个B200上吞吐提升1.23倍。

原文摘要 · Abstract (English)

Long-rollout causal video diffusion has converged on a fixed-size sliding-window KV cache, with recent progress innovating within this layout by changing which tokens occupy the window or how their positions are encoded. The per-head KV layout itself, a dominant contributor to streaming memory and latency, has been mostly left unchanged. In this paper, we present the first study of Multi-Head Latent Attention (MLA) in video diffusion. VideoMLA replaces per-head keys and values with a shared low-rank content latent and a shared decoupled 3D-RoPE positional key, reducing per-token KV memory by 92.7% at every cached layer. We further investigate why MLA succeeds in video diffusion even though the spectral assumption often used to motivate it in language models does not hold: pretrained video attention is not low-rank, with 99%-energy effective rank far above any practical latent dimension. VideoMLA retains quality at compression ratios where direct spectral approximation would predict large reconstruction error. We show that the MLA bottleneck, rather than the pretrained spectrum, determines the effective rank: both spectral and random initialization occupy nearly the full rank budget from initialization, and training preserves this budget while adapting within it. On VBench, VideoMLA matches short-horizon streaming video diffusion baselines, achieves the best overall score at long horizons among evaluated methods, and improves throughput by 1.23x on a single B200.

视频生成扩散模型内存优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。