用归一化流实现端到端视频生成,兼顾质量与效率。
STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flows
- 基于时空潜在空间的全局-局部架构,减少误差累积。
- 采样速度接近扩散模型,生成视频时序一致性强。
- 支持文生视频、图生视频等多种任务,天然具备概率建模能力。
归一化流(NFs)是连续数据的端到端似然生成模型,近年来在图像生成上取得进展。但在视频生成中,由于时空复杂度和计算成本更高,当前主流仍依赖扩散模型。本文提出STARFlow-V,一种基于归一化流的视频生成模型,具有端到端学习、鲁棒因果预测和原生似然估计的优势。基于近期提出的STARFlow,STARFlow-V在时空潜在空间中采用全局-局部架构,将因果依赖限制在全局潜空间,同时保留帧内丰富的局部交互,缓解了标准自回归扩散模型的时间误差累积问题。此外,提出flow-score matching,通过轻量级因果去噪器提升自回归生成的一致性。为提高采样效率,采用视频感知的Jacobi迭代方案,将内部更新转化为可并行的迭代,且不破坏因果性。得益于可逆结构,同一模型原生支持文本到视频、图像到视频及视频到视频生成任务。实验表明,STARFlow-V在视觉保真度和时间一致性方面表现优异,采样吞吐率优于扩散基线。这是首个实证表明归一化流可实现高质量自回归视频生成的工作,为构建世界模型提供了有前景的新方向。代码与生成样本见https://github.com/apple/ml-starflow。
原文摘要 · Abstract (English)
Normalizing flows (NFs) are end-to-end likelihood-based generative models for continuous data, and have recently regained attention with encouraging progress on image generation. Yet in the video generation domain, where spatiotemporal complexity and computational cost are substantially higher, state-of-the-art systems almost exclusively rely on diffusion-based models. In this work, we revisit this design space by presenting STARFlow-V, a normalizing flow-based video generator with substantial benefits such as end-to-end learning, robust causal prediction, and native likelihood estimation. Building upon the recently proposed STARFlow, STARFlow-V operates in the spatiotemporal latent space with a global-local architecture which restricts causal dependencies to a global latent space while preserving rich local within-frame interactions. This eases error accumulation over time, a common pitfall of standard autoregressive diffusion model generation. Additionally, we propose flow-score matching, which equips the model with a light-weight causal denoiser to improve the video generation consistency in an autoregressive fashion. To improve the sampling efficiency, STARFlow-V employs a video-aware Jacobi iteration scheme that recasts inner updates as parallelizable iterations without breaking causality. Thanks to the invertible structure, the same model can natively support text-to-video, image-to-video as well as video-to-video generation tasks. Empirically, STARFlow-V achieves strong visual fidelity and temporal consistency with practical sampling throughput relative to diffusion-based baselines. These results present the first evidence, to our knowledge, that NFs are capable of high-quality autoregressive video generation, establishing them as a promising research direction for building world models. Code and generated samples are available at https://github.com/apple/ml-starflow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。