用快慢双学习机制生成更连贯的长视频，解决远距离帧不一致问题。
SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation
- 引入快慢双速学习：慢学世界动态，快学局部事件记忆
- 在长视频生成中实现0.37次场景切换（基线0.89），FVD降至514
- 适合需要长期一致性与多段记忆的视频生成任务
人类具备互补性学习系统，可将通用世界动态的慢学习与新经验的快速记忆存储相结合。现有视频生成模型主要依赖大规模数据预训练进行慢学习，忽视了对短时记忆存储至关重要的快速学习阶段，导致生成长视频时远距离帧间出现不一致。为此，我们提出SlowFast-VGen，一种面向动作驱动长视频生成的双速学习系统。该方法结合掩码条件视频扩散模型实现世界动态的慢学习，并引入基于时间LoRA模块的推理时快速学习策略，通过局部输入输出实时更新参数，高效存储事件记忆。进一步提出慢-快学习循环算法，将快速学习嵌入慢学习框架，实现多段经验的上下文感知技能学习。为支持慢学习，我们构建了一个包含20万条带语言动作标注视频的大规模数据集。大量实验表明，SlowFast-VGen在多种指标上超越基线，行动驱动视频生成的FVD得分为514（基线782），长视频平均场景切片仅0.37次（基线0.89），显著提升长时程规划任务表现。
原文摘要 · Abstract (English)
Human beings are endowed with a complementary learning system, which bridges the slow learning of general world dynamics with fast storage of episodic memory from a new experience. Previous video generation models, however, primarily focus on slow learning by pre-training on vast amounts of data, overlooking the fast learning phase crucial for episodic memory storage. This oversight leads to inconsistencies across temporally distant frames when generating longer videos, as these frames fall beyond the model's context window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation. Our approach incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference-time fast learning strategy based on a temporal LoRA module. Specifically, the fast learning process updates its temporal LoRA parameters based on local inputs and outputs, thereby efficiently storing episodic memory in its parameters. We further propose a slow-fast learning loop algorithm that seamlessly integrates the inner fast learning loop into the outer slow learning loop, enabling the recall of prior multi-episode experiences for context-aware skill learning. To facilitate the slow learning of an approximate world model, we collect a large-scale dataset of 200k videos with language action annotations, covering a wide range of scenarios. Extensive experiments show that SlowFast-VGen outperforms baselines across various metrics for action-driven video generation, achieving an FVD score of 514 compared to 782, and maintaining consistency in longer videos, with an average of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm significantly enhances performances on long-horizon planning tasks as well. Project Website: https://slowfast-vgen.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。