通过虚拟头机制提升视频扩散模型推理速度，不降质下提速两倍。
Efficient Autoregressive Video Diffusion with Dummy Head
- 设计异构记忆分配与动态头分类，控制不同注意力头的上下文访问。
- 无需训练实现最高2.0倍加速，生成速度达24.3 FPS，质量损失<0.5%。
- 适合追求高效视频生成的开发者与部署场景。
自回归视频扩散模型因具备因果建模和迭代去噪能力而受到关注。本文发现其多头自注意力中约25%的注意力头几乎仅关注当前帧，丢弃其键值缓存仅导致轻微性能下降。基于此，提出Dummy Forcing方法，通过异构内存分配与动态头编程，减少头级上下文冗余，并引入上下文打包技术实现更激进的缓存压缩。该方法无需额外训练，相比基线模型最高可实现2.0倍加速，在24.3 FPS下生成视频，质量损失小于0.5%。
原文摘要 · Abstract (English)
The autoregressive video diffusion model has recently gained considerable research interest due to its causal modeling and iterative denoising. In this work, we identify that the multi-head self-attention in these models under-utilizes historical frames: approximately 25% heads attend almost exclusively to the current frame, and discarding their KV caches incurs only minor performance degradation. Building upon this, we propose Dummy Forcing, a simple yet effective method to control context accessibility across different heads. Specifically, the proposed heterogeneous memory allocation reduces head-wise context redundancy, accompanied by dynamic head programming to adaptively classify head types. Moreover, we develop a context packing technique to achieve more aggressive cache compression. Without additional training, our Dummy Forcing delivers up to 2.0x speedup over the baseline, supporting video generation at 24.3 FPS with less than 0.5% quality drop. Project page is available at https://csguoh.github.io/project/DummyForcing/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。