用对抗训练让视频模型实时生成，边看边改。
Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
- 用单次神经网络计算逐帧生成，高效利用缓存。
- 80亿参数模型在单张H100上实现24帧/秒实时流输出。
- 支持用户交互控制，适合需要即时反馈的应用场景。
现有大规模视频生成模型计算开销大，难以用于实时交互应用。本文提出自回归对抗后训练（AAPT），将预训练的潜在视频扩散模型转化为实时交互式视频生成器。模型通过单一神经函数评估（1NFE）自回归地逐帧生成潜在表示，可实时向用户流式传输结果，并接收交互反馈以生成下一帧。与现有方法不同，本方法探索了对抗训练在自回归生成中的有效性，不仅设计出更高效的单步生成架构并充分利用键值缓存，还实现了学生强迫式训练，有效减少长视频生成中的误差累积。实验表明，我们的8B模型在单张H100上实现736×416分辨率下24帧/秒的实时流式生成，或在8×H100上实现1280×720分辨率下长达一分钟（1440帧）的生成。访问官网：https://seaweed-apt.com/
原文摘要 · Abstract (English)
Existing large-scale video generation models are computationally intensive, preventing adoption in real-time and interactive applications. In this work, we propose autoregressive adversarial post-training (AAPT) to transform a pre-trained latent video diffusion model into a real-time, interactive video generator. Our model autoregressively generates a latent frame at a time using a single neural function evaluation (1NFE). The model can stream the result to the user in real time and receive interactive responses as controls to generate the next latent frame. Unlike existing approaches, our method explores adversarial training as an effective paradigm for autoregressive generation. This not only allows us to design an architecture that is more efficient for one-step generation while fully utilizing the KV cache, but also enables training the model in a student-forcing manner that proves to be effective in reducing error accumulation during long video generation. Our experiments demonstrate that our 8B model achieves real-time, 24fps, streaming video generation at 736x416 resolution on a single H100, or 1280x720 on 8xH100 up to a minute long (1440 frames). Visit our research website at https://seaweed-apt.com/2
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。