提出MAP预训练方法，让混合模型在视觉任务中表现更优。
MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretraining
- 结合掩码自编码与自回归预训练，统一优化混合模型
- 在2D和3D数据集上均达到顶尖性能，超越现有方法
- 适用于需要长序列建模的视觉任务，如视频理解
混合Mamba-Transformer网络近期受到广泛关注。这类网络既能利用Transformer的可扩展性，又能发挥Mamba在长序列建模和计算效率上的优势。然而，如何有效预训练此类混合架构仍是开放问题。现有方法如掩码自编码器（MAE）或自回归（AR）预训练主要针对单一结构，而混合架构的预训练需同时适配Mamba与Transformer模块。为此，我们提出掩码自回归预训练（MAP），用于训练混合Mamba-Transformer视觉骨干网络。该策略融合了MAE与自回归预训练的优势，在统一框架下提升Mamba与Transformer模块的性能。实验表明，经MAP预训练的混合模型显著优于其他预训练策略，达到当前最优水平。我们在2D和3D数据集上验证了方法的有效性，并提供了详细的消融研究以支持各组件设计。代码与检查点已开源。
原文摘要 · Abstract (English)
Hybrid Mamba-Transformer networks have recently garnered broad attention. These networks can leverage the scalability of Transformers while capitalizing on Mamba's strengths in long-context modeling and computational efficiency. However, the challenge of effectively pretraining such hybrid networks remains an open question. Existing methods, such as Masked Autoencoders (MAE) or autoregressive (AR) pretraining, primarily focus on single-type network architectures. In contrast, pretraining strategies for hybrid architectures must be effective for both Mamba and Transformer components. Based on this, we propose Masked Autoregressive Pretraining (MAP) to pretrain a hybrid Mamba-Transformer vision backbone network. This strategy combines the strengths of both MAE and Autoregressive pretraining, improving the performance of Mamba and Transformer modules within a unified paradigm. Experimental results show that the hybrid Mamba-Transformer vision backbone network pretrained with MAP significantly outperforms other pretraining strategies, achieving state-of-the-art performance. We validate the method's effectiveness on both 2D and 3D datasets and provide detailed ablation studies to support the design choices for each component. The code and checkpoints are available at https://github.com/yunzeliu/MAP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。