让分割模型更快更省内存，精度损失极小。
SparseSAM: Structured Sparsification of Activations in Segment Anything Models
- 通过结构化稀疏化同时优化注意力和MLP层，不需重新训练。
- 在0.4密度下仅损失0.004 mIoU，速度提升2倍、内存减少2.8倍。
- 适合追求高效率的部署场景，尤其对资源受限环境友好。
分割一切模型（SAM）虽具备强开放词汇分割能力，但其基于ViT的图像编码器主导了推理延迟与内存占用。现有激活压缩方法如令牌合并虽减少处理令牌数，但引入显著运行时开销，并在高压缩率下导致质量急剧下降。其他方法仅关注稀疏注意力，忽视MLP仍为全连接，限制加速潜力。本文提出SparseSAM，一种无需训练的结构化稀疏化框架，可联合加速注意力与MLP层并保持令牌身份。引入条带排序注意力（Stripe-Sort Attention），采用确定性Z阶排列将密集注意力转为静态硬件友好的稀疏模式，消除动态掩码开销。进一步提出残差一致性MLP，仅将有信息量的令牌送入MLP，其余通过残差路径传递。在四个分割基准上，SparseSAM在0.4密度下仅损失0.004 mIoU，0.3密度下损失0.021 mIoU，相比令牌合并方法准确率损失降低2.10倍，同时实现2倍加速与2.8倍内存减少。
原文摘要 · Abstract (English)
The Segment Anything Model (SAM) achieves strong open-vocabulary segmentation, but its ViT-based image encoders dominate inference latency and memory. Existing activation compression methods, such as token merging, reduce the token length to process, yet introduce non-trivial runtime overhead and encounter catastrophic quality drop under high compression. Other methods applying Sparse Attention focus on attention alone, leaving the MLP fully dense and capping achievable speedup. We propose SparseSAM, a (i) training-free structured sparsification framework that jointly accelerates attention and MLP layers while preserving token identity. SparseSAM introduces (ii) Stripe-Sort Attention, which uses a deterministic Z-order permutation to transform dense attention into static hardware-friendly sparse patterns, eliminating dynamic masking overhead. SparseSAM further introduces a (iii) Residual-Consistency MLP that routes only informative tokens through the MLP while propagating remaining tokens through the residual pathway. Across four segmentation benchmarks, SparseSAM loses only 0.004 mIoU at a 0.4 density and 0.021 mIoU at 0.3, a 2.10x reduction in accuracy loss versus token merging advances, while achieving 2x faster inference and 2.8x memory reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。