arXiv:2410.20502cs.CV2024-10ICLR被引 39

用自回归模型引导扩散模型,高效生成长视频。

ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation

  • 融合自回归模型与扩散Transformer,利用粗粒度时空信息引导生成。
  • 在VBench上8项指标超越OpenSora-V1.2,动态性和美学质量显著提升。
  • 适合需要长视频生成且关注效率与一致性的研究者和开发者。

文本到视频模型近年发展迅速,但受限于数据与算力,实现高质量、高动态性且时序一致的长视频高效生成仍是重大挑战。本文提出ARLON框架,通过将自回归(AR)模型提供的粗粒度空间与长程时序信息融入扩散Transformer(DiT),提升长视频生成效果。具体创新包括:1)采用潜在矢量量化变分自编码器(VQ-VAE)压缩DiT输入潜空间,生成紧凑视觉标记,连接AR与DiT模型,平衡学习复杂度与信息密度;2)设计基于自适应归一化的语义注入模块,将AR模型输出的离散视觉单元嵌入DiT,确保生成过程有效引导;3)为增强对AR推理引入噪声的容忍度,训练时采用带不确定采样的粗粒度潜变量。实验表明,ARLON在VBench选取的11项指标中,有8项优于基线OpenSora-V1.2,动态性与审美质量明显提升,其余3项表现相当,并实现生成加速。同时,在长视频生成任务中达到当前最优性能。文中还分析了推理效率提升机制,并展示通过渐进式文本提示生成长视频的应用实例。演示详见 http://aka.ms/arlon。

原文摘要 · Abstract (English)

Text-to-video models have recently undergone rapid and substantial advancements. Nevertheless, due to limitations in data and computational resources, achieving efficient generation of long videos with rich motion dynamics remains a significant challenge. To generate high-quality, dynamic, and temporally consistent long videos, this paper presents ARLON, a novel framework that boosts diffusion Transformers with autoregressive models for long video generation, by integrating the coarse spatial and long-range temporal information provided by the AR model to guide the DiT model. Specifically, ARLON incorporates several key innovations: 1) A latent Vector Quantized Variational Autoencoder (VQ-VAE) compresses the input latent space of the DiT model into compact visual tokens, bridging the AR and DiT models and balancing the learning complexity and information density; 2) An adaptive norm-based semantic injection module integrates the coarse discrete visual units from the AR model into the DiT model, ensuring effective guidance during video generation; 3) To enhance the tolerance capability of noise introduced from the AR inference, the DiT model is trained with coarser visual latent tokens incorporated with an uncertainty sampling module. Experimental results demonstrate that ARLON significantly outperforms the baseline OpenSora-V1.2 on eight out of eleven metrics selected from VBench, with notable improvements in dynamic degree and aesthetic quality, while delivering competitive results on the remaining three and simultaneously accelerating the generation process. In addition, ARLON achieves state-of-the-art performance in long video generation. Detailed analyses of the improvements in inference efficiency are presented, alongside a practical application that demonstrates the generation of long videos using progressive text prompts. See demos of ARLON at http://aka.ms/arlon.

视频生成扩散模型自回归长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。