用线性注意力替代传统自注意力,实现长视频生成的高效记忆与稳定输出。
Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion

- 引入双分支注意力:局部软硬件注意保留细节,递归线性模块维持固定大小上下文。
- 在75%层使用该结构,速度提升2.26倍,内存减少54%,且时间一致性更优。
- 适合需要长时间流式生成视频的场景,如实时交互式视频创作。
自回归视频扩散模型在流式与交互式视频生成中表现强大,但依赖软硬件注意力导致序列长度上计算复杂度与内存占用呈二次增长,限制了其在长视频上的扩展。现有方法(如稀疏注意力和键值缓存压缩)虽降低每步开销,但仍面临缓存线性增长或不可逆丢弃历史信息的问题,无法解决线性内存增长与流式上下文管理难题。为此,我们提出ARL2(Attend Locally, Remember Linearly),一种混合注意力模块,将二次交叉帧注意力替换为固定大小的递归状态。该模块将自注意力分解为两个分支:帧内软硬件分支用于捕捉空间细节与局部依赖,帧间门控递归线性分支则维护固定大小的状态以支持流式上下文。核心思想是软硬件注意力擅长精细局部交互,而递归状态可实现可控的长程记忆。此设计实现线性时间复杂度与恒定内存消耗,同时提升整体时间一致性。为避免噪声中间状态污染记忆,仅在去噪完成后更新递归状态;为避免帧内信息不对称,所有令牌共享同一预更新状态而非顺序更新。据我们所知,这是首个通过高效两阶段训练将预训练自回归视频扩散模型转化为混合线性注意力架构的工作。当75%的层被替换为混合线性注意力时,模型实现最高达2.26倍的墙钟速度提升与54%的内存减少,同时保持与全软硬件模型相当的质量,并进一步改善时间一致性。
原文摘要 · Abstract (English)
Autoregressive (AR) video diffusion is a powerful paradigm for streaming and interactive video generation. However, its reliance on softmax self-attention leads to quadratic compute complexity in sequence length and memory usage due to key-value caching, which limits its scalability to long video horizons. Existing remedies (e.g., sparse attention and KV-cache compression) reduce per-step cost but still rely on a linearly growing cache or irreversibly discard past context, and thus fail to address linear memory growth and streaming context management. To address this scalability bottleneck, we propose ARL2 (Attend Locally, Remember Linearly), a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state. We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame gated recurrent linear branch that maintains a fixed-size state for streaming context. Our key insight is that softmax attention captures fine-grained local interactions, while a recurrent state provides controllable long-range memory. This design achieves linear-time scaling with constant memory while improving temporal consistency over the full-softmax model. To prevent noisy intermediate states from corrupting memory, we update the recurrent state only after the denoised pass. To avoid within-frame information asymmetry, all tokens share the same pre-update state rather than sequential updates. To the best of our knowledge, this is the first work to convert a pretrained AR video diffusion model into a hybrid linear attention architecture, through an efficient two-stage training scheme for AR video. With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。