不训练也不增加计算量，就能让专家模型在推理时减少一半激活专家数，性能下降极小。
Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models
- 用不同大小的专家集做归一化，分离路由与输出增益影响
- 在35B和397B模型上，专家数减半仅损失0.35~0.55点准确率
- 适合需要高效推理但又不想大幅降性能的部署场景
现代细粒度混合专家（MoE）模型将每个标记路由到少量专家，并对路由器概率进行重归一化。我们发现这种归一化隐式地将专家输出增益与训练时的top-k相关联：推理时减小k不仅改变使用哪些专家，还改变了专家分支的强度。通过激活top-k₁专家，但以top-k₂专家的概率质量进行归一化，引入一个无参数、无需训练、无额外计算开销的整数。在Qwen3.6-35B-A3B上，从8个专家减到4个，在标准归一化下导致MMLU下降4.65分，而使用k₂=16时仅下降0.35分，同时路由专家计算量减半。该结果在更大的Qwen3.5-397B-A17B上复现，从10个减到5个专家，适当参考集下仅损失0.55分。完全移除归一化会导致灾难性后果，表明保持合适的参考概率质量至关重要。分析还显示，困惑度与下游准确率偏好不同的k₂值，警示仅用无标签文本选择压缩设置存在风险。专家身份比权重更重要，且均衡与领域专用路由均留有较少剪枝空间。
原文摘要 · Abstract (English)
Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the training top-$k$: reducing $k$ at inference changes not only which experts are used but also the strength of the expert branch. We separate these effects by activating the top $k_1$ experts while normalizing by the probability mass of the top $k_2$ experts, introducing one integer with no parameters, training, or measurable compute overhead. On Qwen3.6-35B-A3B, reducing from 8 to 4 experts causes a 4.65-point MMLU drop under standard renormalization but only 0.35 points with $k_2=16$, while halving routed-expert compute. The result replicates on the $11\times$ larger Qwen3.5-397B-A17B, where reducing from 10 to 5 experts loses only 0.55 points with an appropriate reference set. Removing renormalization entirely is catastrophic, showing that preserving a suitable reference mass is crucial. We further find that perplexity and downstream accuracy favor different $k_2$, cautioning against selecting MoE compression settings using unlabeled text alone. Analyses also show that expert identity matters substantially more than expert weighting, while balanced and domain-specialized routing leaves limited room for expert pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。