通过插入分隔符，让视觉Mamba模型处理更长的序列，提升图像识别性能。
Separators in Enhancing Autoregressive Pretraining for Vision Mamba
- 在每张图像前加入相同分隔符，区分不同图像输入。
- 输入序列长度提升至原来的4倍，图像尺寸不变，准确率达83.5%。
- 适合希望提升视觉Mamba长序列建模能力的研究者。
状态空间模型Mamba在计算机视觉中崭露头角，因其高效处理长序列任务而备受关注。其固有的因果机制使其特别适用于自回归预训练。然而，现有自回归预训练方法局限于短序列任务，未能充分发挥Mamba处理长序列的优势。为此，本文提出一种新的自回归预训练方法——STAR（Separator for Autoregressive Pretraining），通过在每张图像前插入相同的分隔符来区分不同图像的起始位置。该方法使Vision Mamba的输入序列长度扩大四倍，同时保持原始图像维度。采用该长序列预训练策略，我们的STAR-B模型在ImageNet-1k上达到83.5%的准确率，表现优异，充分体现了利用长程依赖增强视觉模型潜力的可能性。
原文摘要 · Abstract (English)
The state space model Mamba has recently emerged as a promising paradigm in computer vision, attracting significant attention due to its efficient processing of long sequence tasks. Mamba's inherent causal mechanism renders it particularly suitable for autoregressive pretraining. However, current autoregressive pretraining methods are constrained to short sequence tasks, failing to fully exploit Mamba's prowess in handling extended sequences. To address this limitation, we introduce an innovative autoregressive pretraining method for Vision Mamba that substantially extends the input sequence length. We introduce new \textbf{S}epara\textbf{T}ors for \textbf{A}uto\textbf{R}egressive pretraining to demarcate and differentiate between different images, known as \textbf{STAR}. Specifically, we insert identical separators before each image to demarcate its inception. This strategy enables us to quadruple the input sequence length of Vision Mamba while preserving the original dimensions of the dataset images. Employing this long sequence pretraining technique, our STAR-B model achieved an impressive accuracy of 83.5\% on ImageNet-1k, which is highly competitive in Vision Mamba. These results underscore the potential of our method in enhancing the performance of vision models through improved leveraging of long-range dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。