arXiv:2502.18886cs.CLcs.LG2025-02EMNLP被引 7
探索了状态空间模型的剪枝可行性与效果
On Pruning State-Space LLMs
- 将多种剪枝方法适配到状态空间模型结构中
- 部分剪枝方法（如WANDA）能保持模型性能
- 某些剪枝方式会导致性能快速下降，需谨慎选择
近期工作提出状态空间模型（SSMs）作为替代基于Transformer的大型语言模型的高效方案。这些模型能否进一步通过剪枝降低计算成本？我们针对四种基于SSM的LLM，在多个任务上适配并应用了几种剪枝方法。结果发现，这些模型对某些剪枝方法（如WANDA）具有较强鲁棒性，而使用其他方法则会导致性能迅速下降。
原文摘要 · Abstract (English)
Recent work proposed state-space models (SSMs) as an efficient alternative to transformer-based LLMs. Can these models be pruned to further reduce their computation costs? We adapt several pruning methods to the SSM structure, and apply them to four SSM-based LLMs across multiple tasks. We find that such models are quite robust to some pruning methods (e.g. WANDA), while using other methods lead to fast performance degradation.
模型剪枝状态空间模型LLM优化
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。