通过分类注意力头实现视频生成加速，不降质还防闪烁。
HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation
- 按注意力头行为分类，拆分缓存路径，实现高效推理
- 1080P下提速1.95倍，720P下1.62倍，质量与全量一致
- 无需重训练，可直接部署，适合长视频生成场景
自回归视频扩散模型在长视频和流式生成中表现出色，但持续增长的键值缓存使注意力成为主要推理开销，尤其在高分辨率下每帧产生大量标记。现有方法或采用粗粒度启发式策略清除缓存导致帧间闪烁，或需重新训练模型。我们提出HeadCast，一种无需训练、即插即用的加速框架，基于预训练模型注意力头行为稳定且异质的观察。经过短暂预热后，HeadCast在最大噪声步骤进行一次性分类，将每个注意力头归入四类：汇点（Sink）、无效（Dummy）、空间（Spatial）和全局（Global），并将单一键值缓存重构为头特定路径。关键在于保留全局头以维持长时序一致性，避免激进剔除造成的失真。由于空间路径在固定大小网格上运行，其节省随分辨率提升而增加：在主流AR模型上，720P下最高提速1.62倍，1080P下达1.95倍，同时保持VBench评分与全注意力相当，且基本无闪烁。代码已开源。
原文摘要 · Abstract (English)
Autoregressive (AR) video diffusion models have become a promising paradigm for long and streaming video synthesis, but the continuously growing Key-Value (KV) cache makes attention the dominant inference cost, especially at high resolution where each frame contributes many tokens. Existing remedies either evict the cache with coarse heuristics that cause inter-frame flickering, or require model re-training. We propose HeadCast, a training-free, plug-and-play acceleration framework built on the observation that a pre-trained AR model's attention heads exhibit stable, heterogeneous behaviors. After a short warm-up, HeadCast performs a one-time classification at the maximum-noise step that sorts every head into one of four archetypes: Sink, Dummy, Spatial, and Global, and restructures the monolithic KV cache into head-specific pathways. Crucially, it retains the Global heads that preserve the long-range temporal consistency aggressive eviction destroys. Because the Spatial pathway operates on a fixed-size grid, its savings grow with resolution: across state-of-the-art AR models, HeadCast accelerates inference by up to 1.62x at 720P and 1.95x at 1080P, while keeping VBench quality on par with full attention and largely flicker-free. Code is available at https://github.com/sjlgaga/HeadCast .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。