用统一模型实现高效视频生成,突破传统自回归瓶颈。
Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspective
- 提出MM-RoPE,更好建模视频时空特征。
- 采用并行离散扩散,训练效率提升且质量不降。
- 仅用48卡即超越多个主流模型,适合高效视频生成研究者。
自回归大语言模型(LLM)统一了多种语言任务,启发了自回归视频生成的初步探索。现有方法或偏离标准LLM架构,依赖庞大外部文本编码器,或因逐词解码导致极高延迟。本文提出Lumos-1,一种基于LLM的统一自回归视频生成模型,结合高效离散扩散机制。首先,针对视频建模,发现1D RoPE不适用于时空相关性,而简单3D RoPE存在频率谱失衡问题,因此提出MM-RoPE,保留原始文本RoPE的同时,以完整频率谱和缩放3D位置无缝适配视频数据。其次,为契合视频特性并克服逐词解码低效,采用基于掩码的并行离散扩散,结合帧内双向与帧间因果注意力掩码。在此基础上,揭示了因空间信息冗余导致的帧级损失不平衡问题,提出自回归离散扩散强制机制,在训练中引入时序管状掩码,并设计兼容推理掩码策略,避免质量下降。尽管仅使用48块GPU进行预训练与微调,数据有限且采用离散分词器,Lumos-1在GenEval上超越Show-o2,于VBench-I2V上超过COSMOS-Video2World,于VBench-T2V上优于OpenSoraPlan。代码与模型已开源。
原文摘要 · Abstract (English)
Autoregressive large language models (LLMs) have unified a vast range of language tasks, inspiring preliminary efforts in autoregressive (AR) video generation. Existing AR video generators either diverge from standard LLM architectures, depend on bulky external text encoders, or incur prohibitive latency due to next-token decoding. In this paper, we introduce Lumos-1, an LLM-based unified model for AR video generation with efficient discrete diffusion. Firstly, to fit videos with LLMs, we identify that 1D RoPE is ill-suited for visual spatiotemporal correlation modeling, and while demonstrated to be useful, naive 3D RoPE exhibits imbalanced frequency spectra. Therefore, we propose MM-RoPE, which preserves the original textual RoPE while seamlessly accommodating video data with comprehensive frequency spectra and scaled 3D positions. Secondly, to fit the video data's nature and overcome the inefficiency of next-token decoding, we adopt a parallel and mask-based discrete diffusion with the intra-frame bidirectional and inter-frame causal attention masks. Based on this attention mask, we uncover the frame-wise loss imbalance issue caused by spatial information redundancy and propose Autoregressive Discrete Diffusion Forcing, which introduces temporal tube masking during training with a compatible inference-time masking policy to avoid quality degradation. Despite using only 48 GPUs for pre-training and fine-tuning, limited data and a discrete tokenizer, Lumos-1 achieves results surpassing those of Show-o2 on GenEval, COSMOS-Video2World on VBench-I2V, and OpenSoraPlan on VBench-T2V. Code and models are available at https://github.com/alibaba-damo-academy/Lumos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。