arXiv:2602.01801cs.CVcs.AI2026-02中稿 · ICML被引 8

提出FAST-AR框架,让视频扩散模型生成更快更省显存。

Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention

  • 用时序压缩、近似最近邻筛选和稀疏注意力减少冗余计算
  • 推理速度提升5~10倍,显存占用几乎不变
  • 适合长视频生成、交互式神经游戏等需要持续输出的场景

自回归视频扩散模型支持流式生成,适用于长视频合成、视频世界模型和交互式神经游戏引擎。然而,推理时注意力层成为瓶颈:随着生成推进,键值缓存(KV cache)持续增长,导致延迟上升和显存激增,限制了可用时序上下文并损害长程一致性。本文研究自回归视频扩散中的冗余问题,发现三个根源:帧间近似重复的缓存键、缓慢演化的语义查询/键导致大量计算冗余,以及长提示词跨注意力中仅少数标记相关。基于此,提出无需训练的统一注意力框架FAST-AR,包含三个组件:TempCache通过时序对应压缩KV缓存以控制增长;AnnCA利用快速近似最近邻(ANN)匹配筛选帧相关提示词以加速跨注意力;AnnSA通过轻量级ANN将每个查询限制在语义匹配的键上,实现自注意力稀疏化。三者协同降低注意力、计算与内存开销,兼容现有自回归扩散主干和世界模型。实验表明,端到端速度提升5~10倍,视觉质量近乎一致,且长序列推理中吞吐稳定、峰值显存几乎恒定,而基线方法随时间变慢且显存持续上升。

原文摘要 · Abstract (English)

Autoregressive video diffusion models enable streaming generation, opening the door to long-form synthesis, video world models, and interactive neural game engines. However, their core attention layers become a major bottleneck at inference time: as generation progresses, the KV cache grows, causing both increasing latency and escalating GPU memory, which in turn restricts usable temporal context and harms long-range consistency. In this work, we study redundancy in autoregressive video diffusion and identify three persistent sources: near-duplicate cached keys across frames, slowly evolving (largely semantic) queries/keys that make many attention computations redundant, and cross-attention over long prompts where only a small subset of tokens matters per frame. Building on these observations, we propose a unified, training-free attention framework (FAST-AR) for FAST-AutoRegressive diffusion, consisting of three components: TempCache compresses the KV cache via temporal correspondence to bound cache growth; AnnCA accelerates cross-attention by selecting frame-relevant prompt tokens using fast approximate nearest neighbor (ANN) matching; and AnnSA sparsifies self-attention by restricting each query to semantically matched keys, also using a lightweight ANN. Together, these modules reduce attention, compute, and memory and are compatible with existing autoregressive diffusion backbones and world models. Experiments demonstrate up to x5 - x10 end-to-end speedups while preserving near-identical visual quality and, crucially, maintaining stable throughput and nearly constant peak GPU memory usage over long rollouts, where prior methods progressively slow down and suffer from increasing memory usage.

视频生成扩散模型注意力机制高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。