针对长视频生成的错误累积问题，提出按注意力头类型差异化缓存策略。
Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation
- 按注意力头类型分组，设计差异化缓存策略。
- 60秒视频生成质量提升至81.21分，运动与语义更稳定。
- 适合需要高质量长时序生成的研究者与开发者。
自回归视频生成支持流式和开放式长视频合成，但仍受累积误差导致的长期退化影响。现有KV缓存策略通常统一保留历史帧，隐含假设所有注意力头具有相似的历史依赖性。本文重新审视历史帧注意力，发现三类不同头：锚点头需广泛长程上下文，波动头具周期性时间依赖，遮蔽头聚焦初始与邻近帧。基于此，提出金字塔强制（Pyramid Forcing）——一种头感知的分层KV缓存框架，离线识别头类型，分配行为特定缓存策略，并通过高效的不规则缓存注意力支持异构缓存长度。在Self Forcing和Causal Forcing上实验表明，该方法在VBench-Long上持续提升长时序生成质量，60秒Self Forcing得分从77.87提升至81.21，显著增强运动动态、视觉保真度与语义一致性。
原文摘要 · Abstract (English)
Autoregressive video generation enables streaming and open-ended long video synthesis, but still suffers from long-term degradation caused by accumulated errors. Existing KVCache strategies usually apply unified historical-frame retention, implicitly assuming homogeneous historical dependencies across attention heads. We revisit historical-frame attention and reveal three distinct head types: Anchor Heads require broad long-range context, Wave Heads exhibit periodic temporal dependencies, and Veil Heads focus on initial and adjacent frames. Based on this finding, we propose Pyramid Forcing, a head-aware pyramidal KVCache framework that identifies head types offline, assigns behavior-specific cache policies, and supports heterogeneous cache lengths via efficient ragged-cache attention. Experiments on Self Forcing and Causal Forcing show that Pyramid Forcing consistently improves long-horizon generation quality on VBench-Long, increasing the 60-second Self Forcing score from 77.87 to 81.21 while enhancing motion dynamics, visual fidelity, and semantic consistency. Project: https://if-lab-pku.github.io/Pyramid-Forcing/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。