用自回归语言模型生成分钟级长视频，突破现有短时限制。
Loong: Generating Minute-level Long Videos with Autoregressive Language Models
- 将文本与视频令牌统一建模，从头训练自回归模型。
- 通过渐进式训练和损失重加权，解决长视频训练中的损失不平衡问题。
- 适合关注长视频生成、自回归建模的科研与工程人员。
生成内容丰富的分钟级长视频既具吸引力又充满挑战。自回归大语言模型在自然语言处理中已成功生成连贯且长序列的文本，但其在视频生成领域仅限于几秒短片段。本文深入分析了阻碍自回归大语言模型生成长视频的关键挑战。基于观察与分析，提出新模型Loong，可生成分钟级长视频。具体而言，将文本令牌与视频令牌统一为序列，从头训练模型；采用渐进式由短到长的训练策略，并结合损失重加权机制缓解长视频训练中的损失失衡问题；进一步研究推理策略，包括视频令牌重新编码与采样方法，以减少推理过程中的误差累积。实验表明，Loong可在10秒视频上训练，并扩展生成条件于文本提示的分钟级长视频。更多样本见：https://yuqingwang1029.github.io/Loong-video。
原文摘要 · Abstract (English)
It is desirable but challenging to generate content-rich long videos in the scale of minutes. Autoregressive large language models (LLMs) have achieved great success in generating coherent and long sequences of tokens in the domain of natural language processing, while the exploration of autoregressive LLMs for video generation is limited to generating short videos of several seconds. In this work, we conduct a deep analysis of the challenges that prevent autoregressive LLM-based video generators from generating long videos. Based on the observations and analysis, we propose Loong, a new autoregressive LLM-based video generator that can generate minute-long videos. Specifically, we model the text tokens and video tokens as a unified sequence for autoregressive LLMs and train the model from scratch. We propose progressive short-to-long training with a loss re-weighting scheme to mitigate the loss imbalance problem for long video training. We further investigate inference strategies, including video token re-encoding and sampling strategies, to diminish error accumulation during inference. Our proposed Loong can be trained on 10-second videos and be extended to generate minute-level long videos conditioned on text prompts, as demonstrated by the results. More samples are available at: https://yuqingwang1029.github.io/Loong-video.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。