提出MAMA剪枝法，在极强压缩下仍保持大模型性能。
Improved Methods for Model Pruning and Knowledge Distillation
- 结合权重、偏置与强化学习奖励，设计新型剪枝指标。
- 在极端剪枝比例下，性能接近原始模型。
- 适合资源受限场景下的大模型轻量化部署。
模型剪枝是优化大型语言模型（如R1或o3-mini）性能的技术，但现有方法常导致显著性能下降，或需大量重训练与微调。该技术旨在识别并移除人机交互阶段中贡献度低的神经元与连接，目标是获得更小、更快且知识蒸馏后的模型，可几乎等效于未剪枝模型地快速生成内容。本文提出MAMA剪枝法（Movement and Magnitude Analysis），通过固定预训练阶段的权重与偏置，并结合后训练阶段的GRPO奖励作为新的剪枝指标，有效降低模型规模与计算复杂度，同时在极端剪枝水平下仍保持与原模型相当的性能。初步实验表明，该方法在多种剪枝比例及下游自然语言处理任务中均优于或媲美当前最优方法。
原文摘要 · Abstract (English)
Model pruning is a performance optimization technique for large language models like R1 or o3-mini. However, existing pruning methods often lead to significant performance degradation or require extensive retraining and fine-tuning. This technique aims to identify and remove neurons, connections unlikely leading to the contribution during the human-computer interaction phase. Our goal is to obtain a much smaller and faster knowledge distilled model that can quickly generate content almost as good as those of the unpruned ones. We propose MAMA Pruning, short for Movement and Magnitude Analysis, an improved pruning method that effectively reduces model size and computational complexity while maintaining performance comparable to the original unpruned model even at extreme pruned levels. The improved method is based on weights, bias fixed in the pre-training phase and GRPO rewards verified during the post-training phase as our novel pruning indicators. Preliminary experimental results show that our method outperforms and be comparable to state-of-the-art methods across various pruning levels and different downstream computational linguistics tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。