通过对角线解码实现视频生成加速，速度提升10倍且画质不降。
Fast Autoregressive Video Generation with Diagonal Decoding
- 沿时空令牌网格的对角线路径并行生成，打破逐帧顺序依赖。
- 相比传统方法最高提速10倍，长视频生成效率显著提升。
- 无需重新训练，适配多种模型，适合实时视频生成场景。
自回归Transformer在视频生成中表现优异，但逐个令牌的序列解码成为瓶颈，尤其对包含数万个令牌的长视频。本文提出对角线解码（Diagonal Decoding, DiagD），一种无需训练的推理加速算法，利用视频中的时空相关性。该方法沿时空令牌网格的对角线路径生成令牌，可在每帧内并行解码，并在相邻帧间部分重叠。该算法通用性强，适用于多种生成模型与任务，且可灵活调节推理速度与视觉质量的权衡。此外，我们提出一种低成本微调策略，使模型注意力模式与解码顺序对齐，进一步缓解小规模模型的训练-推理差异。多个自回归视频生成模型与数据集上的实验表明，DiagD相比朴素顺序解码最多实现10倍加速，同时保持相近的视觉保真度。
原文摘要 · Abstract (English)
Autoregressive Transformer models have demonstrated impressive performance in video generation, but their sequential token-by-token decoding process poses a major bottleneck, particularly for long videos represented by tens of thousands of tokens. In this paper, we propose Diagonal Decoding (DiagD), a training-free inference acceleration algorithm for autoregressively pre-trained models that exploits spatial and temporal correlations in videos. Our method generates tokens along diagonal paths in the spatial-temporal token grid, enabling parallel decoding within each frame as well as partially overlapping across consecutive frames. The proposed algorithm is versatile and adaptive to various generative models and tasks, while providing flexible control over the trade-off between inference speed and visual quality. Furthermore, we propose a cost-effective finetuning strategy that aligns the attention patterns of the model with our decoding order, further mitigating the training-inference gap on small-scale models. Experiments on multiple autoregressive video generation models and datasets demonstrate that DiagD achieves up to $10\times$ speedup compared to naive sequential decoding, while maintaining comparable visual fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。