通过分组感知剪枝压缩混合语言模型，实现更小体积、更快推理且更准。
Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning
- 设计分组感知剪枝策略，保留SSM模块结构与序列建模能力
- 将80亿参数模型压缩至40亿，训练数据减少40倍，精度反超同类
- 适合追求高效推理的模型部署场景，尤其适用于混合架构
结合注意力机制与状态空间模型（SSMs）的混合语言模型在准确率和运行效率上达到当前最佳水平。近期研究显示，对仅含注意力机制的模型进行压缩与蒸馏，可在极低训练成本下获得更小、更精准的模型。本文探索了对混合架构的压缩效果。提出一种新的分组感知剪枝策略，有效保留SSM模块的结构完整性及其序列建模能力。进一步证明，仅采用传统剪枝无法获得更好的准确率与推理速度，必须针对SSM进行专门优化。我们的压缩方案整合了SSM、前馈网络、嵌入维度及层剪枝，并结合知识蒸馏方式重训，沿用MINITRON技术路径。使用该方法，将Nemotron-H 8B混合模型压缩至40亿参数，训练所需令牌数减少最多达40倍。所获模型在同等规模下超越现有模型精度，同时实现2倍加速推理，显著提升性能-效率权衡边界。
原文摘要 · Abstract (English)
Hybrid LLM architectures that combine Attention and State Space Models (SSMs) achieve state-of-the-art accuracy and runtime performance. Recent work has demonstrated that applying compression and distillation to Attention-only models yields smaller, more accurate models at a fraction of the training cost. In this work, we explore the effectiveness of compressing Hybrid architectures. We introduce a novel group-aware pruning strategy that preserves the structural integrity of SSM blocks and their sequence modeling capabilities. Furthermore, we demonstrate the necessity of such SSM pruning to achieve improved accuracy and inference speed compared to traditional approaches. Our compression recipe combines SSM, FFN, embedding dimension, and layer pruning, followed by knowledge distillation-based retraining, similar to the MINITRON technique. Using this approach, we compress the Nemotron-H 8B Hybrid model down to 4B parameters with up to 40x fewer training tokens. The resulting model surpasses the accuracy of similarly-sized models while achieving 2x faster inference, significantly advancing the Pareto frontier.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。