提出序列令牌合并方法，显著提升视觉Mamba模型效率
Sequential Token Merging: Revisiting Hidden States
- 双向最近邻合并保留序列依赖，保护分类令牌隐藏状态
- 20%令牌减少仅降1.0%准确率，40%减少仅降1.4%
- 适合追求高效视觉Mamba部署的研究者与工程师
视觉Mamba（ViMs）虽实现亚二次复杂度，但其效率受图像分辨率下令牌数量二次增长的制约。现有方法忽略ViMs内在的有限方向性序列依赖（LDSD）——我们分析揭示的关键信息流机制。进一步发现Mamba的选择性扫描可渐进聚合隐藏状态。基于此，提出序列令牌合并（STM）：1）双向最近邻合并，通过对称空间聚合保留序列依赖；2）隐藏状态保护，稳定分类令牌附近的隐藏状态。STM巧妙利用层间损失收敛特性，将时间遗忘转化为稳定性。实验表明，ViM-Ti在令牌减少20%时仅下降1.0%准确率，ViM-S在减少40%时仅下降1.4%。本方法在极低复杂度下达成最先进效率，同时揭示状态空间模型动态新见解。代码即将发布。
原文摘要 · Abstract (English)
Vision Mambas (ViMs) achieve remarkable success with sub-quadratic complexity, but their efficiency remains constrained by quadratic token scaling with image resolution. While existing methods address token redundancy, they overlook ViMs' intrinsic Limited Directional Sequential Dependence (LDSD) - a critical information flow mechanism revealed in our analysis. We further identify Mamba's selective scan enables gradual information aggregation in hidden states. Based on these insights, we propose Sequential Token Merging (STM), featuring: 1) Bidirectional nearest neighbor merging to preserve sequential dependencies through symmetric spatial aggregation, and 2) Hidden states protection to stabilize the hidden states around the class token. STM strategically leverages Mamba's layer-wise loss convergence to convert temporal forgetfulness into stability. Experiments demonstrate STM's superiority: 1.0% accuracy drop for ViM-Ti at 20% token reduction, and only 1.4% degradation for ViM-S at 40% reduction. Our method achieves state-of-the-art efficiency with minimal complexity, while providing new insights into state-space model dynamics. Codes will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。