提出首个无需训练的剪枝框架，让Mamba模型减半参数且不损失性能。
SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot
- 基于二阶敏感度分析，设计层内剪枝策略适配状态空间结构。
- 剪掉50%权重后零样本精度不变，达到当前最优剪枝效果。
- 适用于稀疏化结构，适合部署资源受限的Mamba类大模型。
状态空间语言模型（如Mamba）在保持与Transformer相当性能的同时，实现了线性复杂度推理，但仍包含数十亿参数，阻碍实际部署。现有的一次性剪枝方法针对注意力模块设计，无法适应选择性状态空间模块（SSM）中共享时间步和离散化的状态转移矩阵特性。本文提出SparseSSM，首个无需训练的剪枝框架，将经典最优大脑手术（OBS）扩展至状态空间架构。其分层算法（i）推导出近似二阶显著性评分，聚合跨时间步的海森迹信息；（ii）引入组件敏感性分析指导前馈网络（FFN）剪枝，揭示Mamba架构中的冗余分布；（iii）可轻松扩展至半结构化与结构化稀疏。实验证明，在不进行微调的情况下剪除50%的SSM权重，零样本准确率无损失，成为当前Mamba类大模型的最优剪枝算法。
原文摘要 · Abstract (English)
State-space language models such as Mamba match Transformer quality while permitting linear complexity inference, yet still comprise billions of parameters that hinder deployment. Existing one-shot pruning methods are tailored to attention blocks and fail to account for the time-shared and discretized state-transition matrix at the heart of the selective state-space module (SSM). In this paper, we introduce SparseSSM, the first training-free pruning framework that extends the classic optimal brain surgeon (OBS) framework to state space architectures. Our layer-wise algorithm (i) derives an approximate second-order saliency score that aggregates Hessian-trace information across time steps, (ii) incorporates a component sensitivity analysis to guide feed-forward network (FFN) pruning, which also sheds light on where redundancy resides in mamba architecture, (iii) can be easily extended to semi-structured and structured sparsity. Empirically, we prune 50% of SSM weights without fine-tuning and observe no zero-shot accuracy loss, achieving the current state-of-the-art pruning algorithm for Mamba-based LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。