不依赖蒸馏技术，实现图像视频大模型SSM的高效扩展与鲁棒性提升
StableMamba: Distillation-free Scaling of Large SSMs for Images and Videos
- 提出马尔可夫-注意力交错架构，解决Mamba模型参数扩展难题
- 在ImageNet-1K等三大数据集上性能提升最高达+1.7%
- 显著增强对JPEG压缩等常见伪影的鲁棒性，适合视觉长序列任务
状态空间模型（SSMs）如S4通过将状态空间方法融入深度学习，引入了新型上下文建模方式。然而，其依赖数据无关矩阵导致全局上下文建模能力受限。Mamba模型通过S6选择性扫描算法引入数据相关变体，提升了长序列建模能力。但基于Mamba的架构在参数规模扩展方面存在困难，制约其在视觉任务中的应用。本文针对图像分类与动作识别任务，提出无需知识蒸馏的大型SSM可扩展性解决方案。通过分析Mamba与注意力模型的差异，设计马尔可夫-注意力交错架构，在保持性能的同时显著提升可扩展性、鲁棒性与效率。在ImageNet-1K、Kinetics-400和Something-Something-v2上的实验表明，该方法使当前最优Mamba模型精度提升最高达+1.7%。
原文摘要 · Abstract (English)
State-space models (SSMs), exemplified by S4, have introduced a novel context modeling method by integrating state-space techniques into deep learning. However, they struggle with global context modeling due to their data-independent matrices. The Mamba model addressed this with data-dependent variants via the S6 selective-scan algorithm, enhancing context modeling, especially for long sequences. However, Mamba-based architectures are difficult to scale with respect to the number of parameters, which is a major limitation for vision applications. This paper addresses the scalability issue of large SSMs for image classification and action recognition without requiring additional techniques like knowledge distillation. We analyze the distinct characteristics of Mamba-based and Attention-based models, proposing a Mamba-Attention interleaved architecture that enhances scalability, robustness, and performance. We demonstrate that the stable and efficient interleaved architecture resolves the scalability issue of Mamba-based architectures for images and videos and increases robustness to common artifacts like JPEG compression. Our thorough evaluation on the ImageNet-1K, Kinetics-400 and Something-Something-v2 benchmarks demonstrates that our approach improves the accuracy of state-of-the-art Mamba-based architectures by up to $+1.7$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。