arXiv:2506.14168cs.CVcs.AI2025-06NeurIPS被引 15

用连续标记实现高效视频生成,参数少、训练省、效果优。

VideoMAR: Autoregressive Video Generatio with Continuous Tokens

  • 基于帧间时序因果与空间双向掩码,设计自回归视频生成新框架。
  • 在VBench-I2V上超越当前最优模型,仅需1/200的训练资源。
  • 支持时空外推,推理时通过渐进温度策略减少误差累积。

基于掩码的自回归模型在连续空间中展现出优异的图像生成能力,但在视频生成领域的潜力尚未充分挖掘。本文提出VideoMAR,一种简洁高效的仅解码器自回归视频生成模型,采用连续标记,逐帧逐空间地进行生成。我们首次确立时序因果性与空间双向性为视频自回归模型的基本原则,并引入下一帧扩散损失以融合掩码与视频生成。针对长序列建模开销大、难度高的问题,提出时序短到长课程学习与空间渐进分辨率训练,并在推理时采用渐进温度策略缓解误差累积。此外,VideoMAR复现了语言模型的多项独特能力:因同时支持时序与空间并行缓存,具备高效率;通过3D旋转编码实现时空外推能力。在VBench-I2V基准测试中,VideoMAR超越此前最先进模型Cosmos I2V,同时仅需9.3%的参数、0.5%的训练数据和0.2%的GPU资源。

原文摘要 · Abstract (English)

Masked-based autoregressive models have demonstrated promising image generation capability in continuous space. However, their potential for video generation remains under-explored. In this paper, we propose \textbf{VideoMAR}, a concise and efficient decoder-only autoregressive image-to-video model with continuous tokens, composing temporal frame-by-frame and spatial masked generation. We first identify temporal causality and spatial bi-directionality as the first principle of video AR models, and propose the next-frame diffusion loss for the integration of mask and video generation. Besides, the huge cost and difficulty of long sequence autoregressive modeling is a basic but crucial issue. To this end, we propose the temporal short-to-long curriculum learning and spatial progressive resolution training, and employ progressive temperature strategy at inference time to mitigate the accumulation error. Furthermore, VideoMAR replicates several unique capacities of language models to video generation. It inherently bears high efficiency due to simultaneous temporal-wise KV cache and spatial-wise parallel generation, and presents the capacity of spatial and temporal extrapolation via 3D rotary embeddings. On the VBench-I2V benchmark, VideoMAR surpasses the previous state-of-the-art (Cosmos I2V) while requiring significantly fewer parameters ($9.3\%$), training data ($0.5\%$), and GPU resources ($0.2\%$).

视频生成自回归连续标记高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。