纯Transformer模型实现高效视频预测，性能超越现有方法。
Video Prediction Transformers without Recurrence or Convolution
- 采用门控Transformer结构，无需循环或卷积层。
- 在四个基准测试中达到当前最优准确率与效率。
- 适合需要高性能实时视频预测的工程应用。
视频预测领域曾以基于RNN的ConvLSTM和基于CNN的SimVP为主流。随着ViT的成功，近期研究将ViT引入这两种框架，取得更好性能。然而，我们提出一个根本性问题：是否存在一种更简单且更高效的方法，既能避免RNN的高计算开销，又能解决CNN的有限感受野和泛化能力差的问题？纯Transformer模型在视频预测中能走多远？本文提出PredFormer，一个完全基于门控Transformer的框架。通过全面分析3D注意力机制在视频预测中的表现，实验表明PredFormer在四个标准基准上均达到领先水平。其在精度和效率上的显著提升，凸显了该模型作为真实场景视频预测强基线的巨大潜力。源代码与训练模型将发布于https://github.com/yyyujintang/PredFormer。
原文摘要 · Abstract (English)
Video prediction has witnessed the emergence of RNN-based models led by ConvLSTM, and CNN-based models led by SimVP. Following the significant success of ViT, recent works have integrated ViT into both RNN and CNN frameworks, achieving improved performance. While we appreciate these prior approaches, we raise a fundamental question: Is there a simpler yet more effective solution that can eliminate the high computational cost of RNNs while addressing the limited receptive fields and poor generalization of CNNs? How far can it go with a simple pure transformer model for video prediction? In this paper, we propose PredFormer, a framework entirely based on Gated Transformers. We provide a comprehensive analysis of 3D Attention in the context of video prediction. Extensive experiments demonstrate that PredFormer delivers state-of-the-art performance across four standard benchmarks. The significant improvements in both accuracy and efficiency highlight the potential of PredFormer as a strong baseline for real-world video prediction applications. The source code and trained models will be released at https://github.com/yyyujintang/PredFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。