针对状态空间模型的令牌压缩难题，提出新型融合重要性与相似性的精调压缩方法。
Rethinking Token Reduction for State Space Models
- 结合令牌重要性与相似性，实现层内细粒度令牌压缩
- 在六项基准上使Mamba-2平均准确率提升5.7%至13.1%
- 显著降低计算开销与内存占用，适合部署优化
状态空间模型（SSMs）近年来发展迅速，尤其在支持并行训练和长程依赖建模方面表现突出。以Mamba为代表的架构已扩展至数十亿参数，采用选择性SSM设计。为推动Mamba的广泛应用，其效率优化至关重要。尽管令牌压缩技术可作为训练后简化策略，但直接应用于SSMs会导致性能大幅下降。通过深入分析，我们揭示了现有方法失败的原因及其局限性。为此，提出一种专用于SSMs的统一后训练令牌压缩方法，融合令牌重要性与相似性，实现细粒度的层内压缩。大量实验表明，该方法在六项基准测试中，相较现有方法使Mamba-2平均准确率提升5.7%至13.1%，同时显著降低计算需求与内存占用。
原文摘要 · Abstract (English)
Recent advancements in State Space Models (SSMs) have attracted significant interest, particularly in models optimized for parallel training and handling long-range dependencies. Architectures like Mamba have scaled to billions of parameters with selective SSM. To facilitate broader applications using Mamba, exploring its efficiency is crucial. While token reduction techniques offer a straightforward post-training strategy, we find that applying existing methods directly to SSMs leads to substantial performance drops. Through insightful analysis, we identify the reasons for this failure and the limitations of current techniques. In response, we propose a tailored, unified post-training token reduction method for SSMs. Our approach integrates token importance and similarity, thus taking advantage of both pruning and merging, to devise a fine-grained intra-layer token reduction strategy. Extensive experiments show that our method improves the average accuracy by 5.7% to 13.1% on six benchmarks with Mamba-2 compared to existing methods, while significantly reducing computational demands and memory requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。