压缩后的MoE模型需微调才能恢复性能，小数据微调效果显著。
Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment
- 压缩后加入少量微调，可显著提升模型性能。
- 仅用3000条数据和1轮训练，平均恢复37.3%性能损失。
- 全参数微调比知识蒸馏更省算力，适合资源受限场景。
无重训练的MoE压缩通过剪枝或合并专家降低部署内存，但常将压缩后的检查点视为最终成果。我们认为这一观点不完整：压缩后的MoE检查点更应被视为需小规模后调整的初始化状态。在两个MoE大模型骨干、四种剪枝/合并方法、三种专家保留率及28个基准上，我们比较了小数据预算下语言模型微调（Full FT）与基于教师的知识蒸馏（KD）。仅使用3,000条C4样本和单轮调整，全参数微调平均恢复原始模型与压缩模型间37.3%的性能差距。此外，微调比标准令牌级知识蒸馏更具成本效益，全参数调整在测试范围内提供最佳成本-性能权衡。结果表明，无重训练压缩应搭配小规模后调整，以恢复大量压缩带来的性能损失。
原文摘要 · Abstract (English)
Retraining-free MoE compression reduces deployment memory by pruning or merging experts, but often treats the compressed checkpoint as the final artifact. We argue that this view is incomplete: compressed MoE checkpoints are better understood as compressed initializations that benefit from a tiny post-compression adjustment stage. Across two MoE LLM backbones, four pruning/merging methods, three expert-retention ratios, and 28 benchmarks, we compare LM fine-tuning and teacher-based KD under matched small-data budgets and measured GPU costs. Using only 3,000 C4 examples and a single epoch of adjustment, Full FT recovers 37.3% of the original-to-compressed performance gap on average. Moreover, LM fine-tuning is more cost-effective than standard token-level KD, and full-parameter adjustment gives the strongest cost--recovery trade-off among the tested scopes. These results suggest that retraining-free compression should be paired with small post-compression adjustment to recover a substantial portion of the performance lost during compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。