删掉一半专家,代码模型仍能用,但不同模型需单独测试。
Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding
- 按任务需求剪掉一半专家子网络,保留核心编码能力。
- 在主要代码评测上损失不显著,但非编码能力下降明显。
- 剪枝效果依赖模型结构,不能跨模型通用,需实测验证。
目前最强的开源代码大模型采用混合专家(MoE)架构:大部分模型规模来自大量专家子网络,但每个输入仅激活其中少数。这对普通开发者设备来说过大,且多数专家功能不会被调用。本文对两个不同家族的开源MoE模型(Qwen3.6-35B-A3B 和 Gemma-4-26B-A4B)进行单次剪枝,评估五种策略在代码任务上的表现。结果显示,任一模型均可移除一半专家而未出现统计显著的代码正确率下降,且性能损失集中于非代码任务。然而,最优剪枝策略在两模型间差异明显,说明跨模型经验不可复用。此外,困惑度指标可能误判剪枝后模型性能;轻量微调可恢复约一半损失;只有当量化低于3比特/权重时,剪枝才优于量化。五种尝试推翻该结论的方法均失败,最后一项表明单次基准测试高估了压缩代价,一次执行反馈修复即可消除2比特量化带来的损失。剪枝有效,但必须针对具体任务和模型进行验证。
原文摘要 · Abstract (English)
The strongest open-weight coding models are mixture-of-experts (MoE) networks: most of their size comes from large pools of "expert" subnetworks, of which only a few act on any token. That pool is why these models do not fit on the machines most developers own, yet for a user who only wants coding help, most experts encode abilities that will never be invoked. We ask how many experts can be removed, and which, by pruning two recent open-weight MoE models from different families (Qwen3.6-35B-A3B and Gemma-4-26B-A4B) under five selection strategies, judged the way a user would: by whether the model still writes correct code. Half the experts can be removed from either model with no statistically detectable loss on the primary code benchmark, and the damage lands almost entirely on abilities outside coding, the intended trade. But the winning strategy flips between the two models, so a recipe validated on one family cannot be assumed to work on another. We further show that perplexity, the metric much of the pruning literature leans on, can rate a broken model above an intact one; that a lightweight fine-tune recovers about half of what aggressive pruning loses; and that against quantizing the full model to the same memory, pruning wins only where quantization would have to drop below 3 bits per weight. Five attempts to overturn that crossover, with failure criteria fixed in advance (better calibration, guarded selection, causal expert importance, failure attribution, and an agentic evaluation letting each model repair its failures from execution feedback), all leave it standing; the last shows single-shot benchmarks overstate compression penalties broadly, as one repair turn erases the 2-bit quantization penalty entirely. Expert pruning works, but it demands per-model validation on the task the model will actually serve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。