通过分层规划实现高质量长视频并行生成
Macro-from-Micro Planning for High-Quality and Parallelized Autoregressive Long Video Generation
- 分两阶段规划:先局部预测关键帧,再全局串联保证连贯性
- 生成过程可并行化,显著提升长视频合成效率
- 适合需要长时序一致性的视频生成任务
当前自回归扩散模型在视频生成上表现优异,但通常局限于短时长。理论分析表明,自回归建模易因误差累积导致时间漂移,并阻碍长视频合成的并行化。为此,我们提出一种新的‘规划-填充’框架,即宏从微规划(MMPL),用于长视频生成。该框架通过两级层次化规划:微观规划在每个短片段内预测稀疏未来关键帧,提供运动与外观先验以指导高质量片段生成;宏观规划则通过自回归链式扩展片段内关键帧规划至全视频,确保跨片段长期一致性。随后,基于MMPL的内容填充并行生成各片段间所有中间帧,实现自回归生成的高效并行。进一步通过自适应负载调度优化计算分配,平衡GPU负载,加速生成。大量实验表明,本方法在质量和稳定性上均优于现有长视频生成模型。生成视频及对比结果详见项目主页。
原文摘要 · Abstract (English)
Current autoregressive diffusion models excel at video generation but are generally limited to short temporal durations. Our theoretical analysis indicates that the autoregressive modeling typically suffers from temporal drift caused by error accumulation and hinders parallelization in long video synthesis. To address these limitations, we propose a novel planning-then-populating framework centered on Macro-from-Micro Planning (MMPL) for long video generation. MMPL sketches a global storyline for the entire video through two hierarchical stages: Micro Planning and Macro Planning. Specifically, Micro Planning predicts a sparse set of future keyframes within each short video segment, offering motion and appearance priors to guide high-quality video segment generation. Macro Planning extends the in-segment keyframes planning across the entire video through an autoregressive chain of micro plans, ensuring long-term consistency across video segments. Subsequently, MMPL-based Content Populating generates all intermediate frames in parallel across segments, enabling efficient parallelization of autoregressive generation. The parallelization is further optimized by Adaptive Workload Scheduling for balanced GPU execution and accelerated autoregressive video generation. Extensive experiments confirm that our method outperforms existing long video generation models in quality and stability. Generated videos and comparison results are in our project page.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。