通过剪枝冗余激活提升视觉状态空间模型推理速度
QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Models
- 训练后剪枝空间激活，用近邻插值恢复维度
- ImageNet-1K上提速11%且准确率下降不足0.9%
- 适配医疗图像等领域的SNN模型，部署即用
状态空间模型（SSMs）通过线性递归将Transformer的二次复杂度降低。最近，VMamba作为基于SSM的视觉骨干网络表现强劲，但仍受四向扫描中空间冗余的限制。我们提出QuarterMap，一种训练后激活剪枝方法，在扫描前移除冗余空间激活，并通过最近邻插值恢复维度。该方法无需重训练即可提升吞吐量。在ImageNet-1K上，QuarterMap使VMamba提速高达11%，准确率下降不足0.9%，在ADE20K分割任务中也取得类似增益。此外，我们在共享相同四向扫描结构的领域专用模型MedMamba上验证了QuarterMap，其在多个医学图像任务中持续提升吞吐量并保持准确率。相比ToMe等令牌合并方法，QuarterMap专为SSMs设计，避免了昂贵的合并-还原操作。本方法为部署阶段效率优化提供了即插即用工具，且不损害迁移能力。
原文摘要 · Abstract (English)
State space models (SSMs) reduce the quadratic complexity of transformers by leveraging linear recurrence. Recently, VMamba has emerged as a strong SSM-based vision backbone, yet remains bottlenecked by spatial redundancy in its four-directional scan. We propose QuarterMap, a post-training activation pruning method that removes redundant spatial activations before scanning and restores dimensions via nearest-neighbor upsampling. Our method improves throughput without retraining. On ImageNet-1K, QuarterMap achieves up to 11% speedup on VMamba with less than 0.9% accuracy drop, and yields similar gains on ADE20K segmentation. Beyond VMamba, we validate QuarterMap on MedMamba, a domain-specific model that shares the same four-directional scanning structure, where it consistently improves throughput while preserving accuracy across multiple medical imaging tasks. Compared to token merging methods like ToMe, QuarterMap is tailored for SSMs and avoids costly merge-unmerge operations. Our method offers a plug-and-play tool for deployment-time efficiency without compromising transferability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。