无需向量量化，用新方法实现高效视频生成。
Autoregressive Video Generation without Vector Quantization
- 将视频生成重构为帧内与帧间无量化自回归建模。
- 0.6B参数模型在速度、画质和流畅性上超越同类方法。
- 支持长视频生成和零样本应用，适合通用视频生成任务。
本文提出一种新型自回归视频生成方法，实现高效建模。通过将视频生成问题重新定义为无向量量化的时间帧级与空间集级自回归预测，该方法保持了GPT类模型的因果性以支持灵活上下文推理，同时利用单帧内双向建模提升效率。我们训练了一个名为NOVA的新模型，其在数据效率、推理速度、视觉保真度和视频流畅性方面均优于先前自回归视频模型，且仅需0.6B参数。此外，NOVA在文本到图像生成任务中超越主流图像扩散模型，训练成本显著更低。该模型在更长视频长度下泛化良好，并支持多种零样本应用，统一于单一模型中。代码与模型已公开于https://github.com/baaivision/NOVA。
原文摘要 · Abstract (English)
This paper presents a novel approach that enables autoregressive video generation with high efficiency. We propose to reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction and spatial set-by-set prediction. Unlike raster-scan prediction in prior autoregressive models or joint distribution modeling of fixed-length tokens in diffusion models, our approach maintains the causal property of GPT-style models for flexible in-context capabilities, while leveraging bidirectional modeling within individual frames for efficiency. With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity, i.e., 0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models in text-to-image generation tasks, with a significantly lower training cost. Additionally, NOVA generalizes well across extended video durations and enables diverse zero-shot applications in one unified model. Code and models are publicly available at https://github.com/baaivision/NOVA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。