用1万亿视觉标记训练自回归视频模型，效果媲美主流方法。
An Empirical Study of Autoregressive Pre-training from Videos
- 将视频转为视觉标记序列，用Transformer自回归预测
- 在图像识别、视频分类等任务上达到竞争力表现
- 模型规模扩展规律与语言模型相似，适合大规模视频预训练
我们对从视频中进行自回归预训练进行了实证研究。为此，我们构建了一系列自回归视频模型，称为Toto。将视频视为视觉标记序列，训练Transformer模型自回归预测未来标记。模型在包含超过1万亿视觉标记的多样化视频与图像数据集上进行预训练。探索了不同架构、训练和推理设计选择。在图像识别、视频分类、目标追踪和机器人等多个下游任务上评估了学习到的视觉表征。结果表明，尽管归纳偏置极少，自回归预训练在所有基准测试中均表现出色。最后发现，扩大视频模型规模时，其缩放曲线与语言模型类似，但速率不同。更多信息见https://brjathu.github.io/toto/
原文摘要 · Abstract (English)
We empirically study autoregressive pre-training from videos. To perform our study, we construct a series of autoregressive video models, called Toto. We treat videos as sequences of visual tokens and train transformer models to autoregressively predict future tokens. Our models are pre-trained on a diverse dataset of videos and images comprising over 1 trillion visual tokens. We explore different architectural, training, and inference design choices. We evaluate the learned visual representations on a range of downstream tasks including image recognition, video classification, object tracking, and robotics. Our results demonstrate that, despite minimal inductive biases, autoregressive pre-training leads to competitive performance across all benchmarks. Finally, we find that scaling our video models results in similar scaling curves to those seen in language models, albeit with a different rate. More details at https://brjathu.github.io/toto/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。