为Mamba模型设计高效剪枝方法，70%参数减少仍保95%性能。
Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments
- 结合权重大小与梯度信息，精准识别冗余参数。
- 迭代渐进剪枝，实现70%参数压缩且性能损失<5%。
- 适合边缘设备部署，提升Mamba在资源受限场景的实用性。
状态空间模型（SSMs），特别是Mamba架构，已成为序列建模的有力替代方案，具备线性时间复杂度和优异的跨任务表现。然而其庞大的参数量给资源受限环境的部署带来挑战。本文提出一种针对Mamba模型的新型无结构剪枝框架，可在保持超过95%原始性能的前提下实现高达70%的参数缩减。该方法融合三项创新：(1) 基于梯度感知的幅度剪枝，结合权重大小与梯度信息识别低重要性参数；(2) 迭代式剪枝策略，逐步提升稀疏度以维持模型稳定性；(3) 全局剪枝策略，优化模型整体参数分配。在WikiText-103、Long Range Arena及ETT时间序列数据集上的大量实验表明，该方法显著提升效率且性能下降极小。对剪枝影响的分析揭示了Mamba架构的冗余性与鲁棒性，为资源受限环境中的实际部署提供了支持，并拓展了Mamba的应用范围。
原文摘要 · Abstract (English)
State-space models (SSMs), particularly the Mamba architecture, have emerged as powerful alternatives to Transformers for sequence modeling, offering linear-time complexity and competitive performance across diverse tasks. However, their large parameter counts pose significant challenges for deployment in resource-constrained environments. We propose a novel unstructured pruning framework tailored for Mamba models that achieves up to 70\% parameter reduction while retaining over 95\% of the original performance. Our approach integrates three key innovations: (1) a gradient-aware magnitude pruning technique that combines weight magnitude and gradient information to identify less critical parameters, (2) an iterative pruning schedule that gradually increases sparsity to maintain model stability, and (3) a global pruning strategy that optimizes parameter allocation across the entire model. Through extensive experiments on WikiText-103, Long Range Arena, and ETT time-series benchmarks, we demonstrate significant efficiency gains with minimal performance degradation. Our analysis of pruning effects on Mamba's components reveals critical insights into the architecture's redundancy and robustness, enabling practical deployment in resource-constrained settings while broadening Mamba's applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。