解决长视频生成中的错误累积问题,实现高质量、连贯的超长视频合成。
Towards Error-Free Long Video Generation

- 用扩散模型分段生成视频,通过因果注意力连接片段保持时间一致性。
- 在长视频数据上微调后,生成分钟级视频仍保持身份一致性和动态细节。
- 采用键值缓存和截断修正流技术,有效控制内存并减少误差传播。
近期视频生成技术已可实现分钟级视频合成,但长视频生成仍面临错误累积、属性漂移及长视频数据稀缺等挑战。本文提出一种无限长度视频生成框架,聚焦解决上述问题,能够生成高质量、动态丰富且身份一致的单次生成长视频。首先,基于大规模短视频数据微调扩散模型作为视频扩展模型,实现自回归式的时间连贯片段生成;受大语言模型启发,引入片段间的因果注意力机制,在长视频数据上进一步微调模型:同一片段内使用双向注意力,片段间则采用单向注意力。该设计结合现代扩散模型优势,同时保留长期上下文信息,有效缓解错误累积与属性漂移。为提升推理时的内存效率,采用键值(KV)缓存机制维持恒定内存占用;此外引入截断修正流(T-RFlow)技术,进一步抑制误差累积。实验表明,本方法在真实感与连贯性方面建立新基准,显著提升分钟级视频生成质量。
原文摘要 · Abstract (English)
Recent advances in video generation have made minute-level synthesis possible; however, generating long videos remains challenging due to error accumulation, attribute drift, and the limited availability of long video data. In this paper, we introduce an infinite-length video generation framework that focusing on addressing these issues and produces high-quality, dynamic, and identity-consistent single-shot long videos. We first finetune a diffusion model as a video extension model on large-scale short video data to autoregressively generate temporally coherent clips. Inspired by the success of large language models (LLMs), we adopt causal attention computation between clips to further finetune this model on long video data. In this way, the tokens in one clip (short video) are computed by bidirectional attention while tokens among clips are computed by unidirectional attention. This design leverages the strengths of modern diffusion models while preserving long-term context information, effectively mitigating error accumulation and attribute drift. To achieve memory efficiency during inference, we adopt a key-value (KV) caching mechanism to maintain a constant KV memory. Furthermore, we introduce truncation-rectified flow (T-RFlow) technique to further suppress error accumulation. Experimental results demonstrate the effectiveness of our method. Our framework establishes a new benchmark for realistic and coherent minute-level video synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。