用历史画面当记忆，让长视频生成更连贯
Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval
- 把过往画面直接存为记忆，输入时拼接预测帧
- 通过视角重叠筛选相关画面，计算量大幅降低
- 支持开放场景泛化，适合交互式长视频生成
近期交互式视频生成进展显著，但长视频生成中缺乏场景一致性记忆能力。本文提出Context-as-Memory，将历史上下文作为记忆用于视频生成。其包含两项设计：(1) 以帧形式存储上下文，无需额外后处理；(2) 在输入端沿帧维度拼接上下文与待预测帧进行条件控制，无需外部控制模块。为应对全部历史上下文带来的巨大计算开销，提出记忆检索模块，通过判断相机位姿的视场（FOV）重叠来筛选真正相关的上下文帧，显著减少候选帧数量且信息损失小。实验表明，该方法在交互式长视频生成中优于当前最优模型，甚至能在训练未见的开放域场景中有效泛化。
原文摘要 · Abstract (English)
Recent advances in interactive video generation have shown promising results, yet existing approaches struggle with scene-consistent memory capabilities in long video generation due to limited use of historical context. In this work, we propose Context-as-Memory, which utilizes historical context as memory for video generation. It includes two simple yet effective designs: (1) storing context in frame format without additional post-processing; (2) conditioning by concatenating context and frames to be predicted along the frame dimension at the input, requiring no external control modules. Furthermore, considering the enormous computational overhead of incorporating all historical context, we propose the Memory Retrieval module to select truly relevant context frames by determining FOV (Field of View) overlap between camera poses, which significantly reduces the number of candidate frames without substantial information loss. Experiments demonstrate that Context-as-Memory achieves superior memory capabilities in interactive long video generation compared to SOTAs, even generalizing effectively to open-domain scenarios not seen during training. The link of our project page is https://context-as-memory.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。