用新出现的内容决定记忆,让长视频生成更连贯。
RECAP-Forcing: Retaining Content Appearances for Long Video Generation

- 按内容首次出现时保留记忆,而非按时间新旧
- 在多个基线模型上提升视觉质量和语义一致性
- 无需训练、无额外参数,适合长视频生成场景
长序列自回归视频生成面临核心记忆挑战:有限注意力窗口下,模型需决定保留历史中哪些信息。现有方法按时间组织记忆,保留近期帧,压缩或丢弃较早内容。本文提出RECAP-Forcing,按外观新颖性组织记忆。长视频不仅是帧的序列,更是随时间演化的角色、物体与场景集合,其身份需保持一致。我们保留新出现内容(如进入的角色、重新可见区域、新引入场景)首次显现时的键值缓存,以新颖性优先于新近性。记忆规模应随新增内容量增长,而非视频长度。这种基于外观的记忆结构使长期一致性成为显式属性。本框架统一两种机制:视频初期,所有内容新颖,使用注意力汇点保存初始场景;视频演化中,基于光流的新颖性银行选择性保留新揭示内容。作为无需训练的推理方法,无额外可学习参数,RECAP-Forcing在多个强基线上持续提升视觉质量与语义保真度,优于现有记忆方法。
原文摘要 · Abstract (English)
Long autoregressive video generation faces a fundamental memory challenge: with a finite attention window, a model must decide which information from an ever-expanding history to retain. Existing methods organize memory temporally, preserving recent frames while compressing or discarding older ones. We instead propose RECAP-Forcing, organizing memory by appearance novelty. A long video is not merely a sequence of frames, but an evolving cast of subjects, objects, and scenes whose identities must remain consistent over time. We organize memory by retaining the KV cache associated with newly appearing content--such as entering subjects, disoccluded regions, and newly introduced scenes--at the moment it first becomes visible, prioritizing novelty over recency. Memory should scale with the amount of newly introduced content, rather than with video length. This appearance-indexed memory makes long-range consistency an explicit property of the memory structure. Our framework unifies two mechanisms under this single principle. At the beginning of a video, when all visible content is novel, an attention sink preserves the initial scene. As the video evolves, an optical-flow-based novelty bank extends the same principle by selectively retaining newly revealed content. As a training-free inference method with no additional learnable parameters, RECAP-Forcing consistently improves visual quality and semantic fidelity across multiple strong baselines and outperforms existing memory methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。