融合自回归与扩散模型,实现灵活异步视频生成
AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion

- 用扩散机制统一训练与推理,减少误差累积
- 支持可变长度视频生成,保持时间连贯性
- 设计专用调度器,适配同步与异步生成场景
视频生成需合成视觉真实且时间连贯的帧序列。现有方法多采用异步自回归模型或同步扩散模型,前者存在训练与推理不一致导致的误差累积问题,后者受限于固定序列长度。为此,我们提出自回归扩散(AR-Diffusion)模型,结合自回归与扩散模型优势,实现灵活异步视频生成。该方法在训练和推理中均通过扩散逐步破坏视频帧,降低两阶段差异;借鉴自回归生成思想,引入非递减污染时间步约束,确保早期帧比后续帧更清晰;配合时间因果注意力机制,支持任意长度视频生成并维持时间一致性。此外,设计两种专用时间步调度器:FoPP调度器用于训练时均衡采样,AD调度器用于推理时灵活调整时间步差,兼容同步与异步生成。大量实验表明,该方法在四个挑战性基准上达到竞争力甚至领先水平。
原文摘要 · Abstract (English)
The task of video generation requires synthesizing visually realistic and temporally coherent video frames. Existing methods primarily use asynchronous auto-regressive models or synchronous diffusion models to address this challenge. However, asynchronous auto-regressive models often suffer from inconsistencies between training and inference, leading to issues such as error accumulation, while synchronous diffusion models are limited by their reliance on rigid sequence length. To address these issues, we introduce Auto-Regressive Diffusion (AR-Diffusion), a novel model that combines the strengths of auto-regressive and diffusion models for flexible, asynchronous video generation. Specifically, our approach leverages diffusion to gradually corrupt video frames in both training and inference, reducing the discrepancy between these phases. Inspired by auto-regressive generation, we incorporate a non-decreasing constraint on the corruption timesteps of individual frames, ensuring that earlier frames remain clearer than subsequent ones. This setup, together with temporal causal attention, enables flexible generation of videos with varying lengths while preserving temporal coherence. In addition, we design two specialized timestep schedulers: the FoPP scheduler for balanced timestep sampling during training, and the AD scheduler for flexible timestep differences during inference, supporting both synchronous and asynchronous generation. Extensive experiments demonstrate the superiority of our proposed method, which achieves competitive and state-of-the-art results across four challenging benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。