用混合架构提升视频自回归预训练的可扩展性与效率
VideoMAP: Toward Scalable Mamba-based Video Autoregressive Pretraining
- 4:1 Mamba与Transformer混合结构，平衡计算与容量
- 帧级掩码自回归预训练，显著提升大模型性能
- 样本高效，适合做多模态大模型的视觉编码器
基于Mamba的视频理解架构虽具计算高效性和竞争力，但存在过拟合问题，制约其可扩展性。为此，我们提出VideoMAP，一种融合Mamba与Transformer的混合框架，采用4:1的Mamba-to-Transformer比例，有效平衡计算开销与模型容量。结合提出的帧级掩码自回归预训练策略，VideoMAP在扩大模型规模时实现显著性能提升。实验表明，其在Kinetics-400、Something-Something V2、Breakfast和COIN等多个数据集上均优于现有方法。此外，VideoMAP具备出色的样本效率，且可作为多模态大语言模型的视觉编码器，降低内存占用并支持更长视频序列处理。代码已开源。
原文摘要 · Abstract (English)
Recent Mamba-based architectures for video understanding demonstrate promising computational efficiency and competitive performance, yet struggle with overfitting issues that hinder their scalability. To overcome this challenge, we introduce VideoMAP, a Hybrid Mamba-Transformer framework featuring a novel pre-training approach. VideoMAP uses a 4:1 Mamba-to-Transformer ratio, effectively balancing computational cost and model capacity. This architecture, combined with our proposed frame-wise masked autoregressive pre-training strategy, delivers significant performance gains when scaling to larger models. Additionally, VideoMAP exhibits impressive sample efficiency, significantly outperforming existing methods with less training data. Experiments show that VideoMAP outperforms existing models across various datasets, including Kinetics-400, Something-Something V2, Breakfast, and COIN. Furthermore, we demonstrate the potential of VideoMAP as a visual encoder for multimodal large language models, highlighting its ability to reduce memory usage and enable the processing of longer video sequences. The code is open-source at https://github.com/yunzeliu/MAP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。