解决长视频生成中记忆丢失问题,实现高一致性实时生成。
Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation
- 将记忆压缩与帧生成分离,用轻量缓存保存历史信息
- 在长视频生成中显著提升场景一致性,性能优于传统方法
- 适合需要长期一致性的交互式视频生成场景
帧级自回归(frame-AR)模型已取得显著进展,可实现接近双向扩散模型的实时视频生成,成为交互式世界模型和游戏引擎的基础。然而,现有长视频生成方法多依赖窗口注意力,简单丢弃窗口外的历史上下文,导致灾难性遗忘和场景不一致;而保留完整历史则带来高昂内存开销。为此,我们提出记忆-生成框架MAG,将记忆压缩与帧生成解耦为独立任务:训练一个记忆模型将历史信息压缩为紧凑的键值缓存(KV cache),再由独立生成模型利用该压缩表示合成后续帧。此外,我们构建MAG-Bench以严格评估历史记忆保持能力。大量实验表明,MAG在保持标准视频生成基准竞争力的同时,显著提升了长视频中的场景一致性。
原文摘要 · Abstract (English)
Frame-level autoregressive (frame-AR) models have achieved significant progress, enabling real-time video generation comparable to bidirectional diffusion models and serving as a foundation for interactive world models and game engines. However, current approaches in long video generation typically rely on window attention, which naively discards historical context outside the window, leading to catastrophic forgetting and scene inconsistency; conversely, retaining full history incurs prohibitive memory costs. To address this trade-off, we propose Memorize-and-Generate (MAG), a framework that decouples memory compression and frame generation into distinct tasks. Specifically, we train a memory model to compress historical information into a compact KV cache, and a separate generator model to synthesize subsequent frames utilizing this compressed representation. Furthermore, we introduce MAG-Bench to strictly evaluate historical memory retention. Extensive experiments demonstrate that MAG achieves superior historical scene consistency while maintaining competitive performance on standard video generation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。