实现30+帧每秒的自回归视频生成,突破实时交互瓶颈。
Playing with Transformer at 30+ FPS via Next-Frame Diffusion
- 采用分块因果注意力与并行生成,提升自回归视频采样效率。
- 结合一致性蒸馏与推测采样,在A100上实现310M模型超30FPS生成。
- 适合需要低延迟视频生成的交互应用,如游戏与实时渲染。
自回归视频模型在生成交互式内容和支持任意时长流媒体方面优于双向扩散模型。本文提出下一帧扩散(NFD),一种基于块级因果注意力的自回归扩散变换器,支持逐帧并行生成与迭代采样。然而,扩散采样高计算成本及自回归生成硬件效率低仍是实时生成的主要挑战。为此,我们引入两项创新:(1) 将一致性蒸馏扩展至视频领域,并针对视频模型进行适配,实现少步数高效推理;(2) 观察到相邻帧常共享相同动作输入,提出推测采样:模型用当前动作输入生成后续几帧,若动作变化则丢弃推测结果。在大规模动作条件视频生成基准上,NFD在视觉质量与采样效率上均超越自回归基线。首次在A100 GPU上使用310M模型实现超过30帧每秒的自回归视频生成。
原文摘要 · Abstract (English)
Autoregressive video models offer distinct advantages over bidirectional diffusion models in creating interactive video content and supporting streaming applications with arbitrary duration. In this work, we present Next-Frame Diffusion (NFD), an autoregressive diffusion transformer that incorporates block-wise causal attention, enabling iterative sampling and efficient inference via parallel token generation within each frame. Nonetheless, achieving real-time video generation remains a significant challenge for such models, primarily due to the high computational cost associated with diffusion sampling and the hardware inefficiencies inherent to autoregressive generation. To address this, we introduce two innovations: (1) We extend consistency distillation to the video domain and adapt it specifically for video models, enabling efficient inference with few sampling steps; (2) To fully leverage parallel computation, motivated by the observation that adjacent frames often share the identical action input, we propose speculative sampling. In this approach, the model generates next few frames using current action input, and discard speculatively generated frames if the input action differs. Experiments on a large-scale action-conditioned video generation benchmark demonstrate that NFD beats autoregressive baselines in terms of both visual quality and sampling efficiency. We, for the first time, achieves autoregressive video generation at over 30 Frames Per Second (FPS) on an A100 GPU using a 310M model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。