用分层流匹配提升视频生成效率，支持10秒768p高清视频生成。
Pyramidal Flow Matching for Efficient Video Generative Modeling
- 设计分层流匹配框架，仅末层处理全分辨率，降低计算负担。
- 实现端到端训练，20.7k A100小时可生成5-10秒768p/24FPS视频。
- 适合追求高效高质视频生成的研究者与开发者使用。
视频生成需建模庞大的时空空间，计算与数据开销巨大。现有方法多采用级联架构避免全分辨率直接训练，但各阶段独立优化导致知识无法共享且灵活性差。本文提出统一的分层流匹配算法，将原始去噪轨迹重构成多级金字塔结构，仅末级在全分辨率运行，显著提升效率。通过精心设计，各层级流可相互连通以保持连续性；并引入时序金字塔实现自回归视频生成，压缩全分辨率历史。整个框架可端到端优化，仅用一个统一的Diffusion Transformer（DiT）。大量实验表明，该方法可在20.7k A100 GPU小时训练下生成高质量5秒（最长可达10秒）768p分辨率、24帧/秒的视频。代码与模型已开源于https://pyramid-flow.github.io。
原文摘要 · Abstract (English)
Video generation requires modeling a vast spatiotemporal space, which demands significant computational resources and data usage. To reduce the complexity, the prevailing approaches employ a cascaded architecture to avoid direct training with full resolution latent. Despite reducing computational demands, the separate optimization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video generative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, we craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models are open-sourced at https://pyramid-flow.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。