用扩散模型生成视频，实现短时与长时预测的统一建模。
Video-GPT via Next Clip Diffusion
- 基于历史清晰片段，逐帧去噪生成新视频片段。
- 在Physics-IQ基准上达34.97分，超越现有方法。
- 通用性强，可适配6类主流视频生成与理解任务。
GPT在自然语言处理中表现出色，但语言序列难以描述视觉世界中的时空细节。相比之下，视频序列更擅长捕捉这些信息。受此启发，本文提出一种简洁的Video-GPT，将视频视为视觉世界的新语言。类比GPT的下一个词预测，引入新型“下一片段扩散”预训练范式：通过自回归方式，根据历史清晰片段逐步去噪生成噪声片段。该方法同时支持短时生成与长时预测。大量实验表明，Video-GPT在视频预测任务上达到领先性能（Physics-IQ基准：Video-GPT 34.97 vs. Kling 23.64 vs. Wan 20.89）。此外，其在6个主流视频任务（包括生成与理解）中均表现优异，展现出强大的下游泛化能力。
原文摘要 · Abstract (English)
GPT has shown its remarkable success in natural language processing. However, the language sequence is not sufficient to describe spatial-temporal details in the visual world. Alternatively, the video sequence is good at capturing such details. Motivated by this fact, we propose a concise Video-GPT in this paper by treating video as new language for visual world modeling. By analogy to next token prediction in GPT, we introduce a novel next clip diffusion paradigm for pretraining Video-GPT. Different from the previous works, this distinct paradigm allows Video-GPT to tackle both short-term generation and long-term prediction, by autoregressively denoising the noisy clip according to the clean clips in the history. Extensive experiments show our Video-GPT achieves the state-of-the-art performance on video prediction, which is the key factor towards world modeling (Physics-IQ Benchmark: Video-GPT 34.97 vs. Kling 23.64 vs. Wan 20.89). Moreover, it can be well adapted on 6 mainstream video tasks in both video generation and understanding, showing its great generalization capacity in downstream. The project page is at https://zhuangshaobin.github.io/Video-GPT.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。