让低效的MoE模型更省力，只改0.7%参数就接近全量微调。
EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning
- 根据路由概率动态剪枝不重要的专家，增强重要专家的参数能力。
- 在相同参数预算下，性能超越LoRA，仅更新0.55%-0.72%参数。
- 适合资源受限场景，尤其对大尺寸MoE模型微调有显著优势。
Mixture-of-Experts (MoE) 模型虽具高效扩展性，但适应成本仍高，源于冗余专家和均匀参数分配。现有参数高效微调（PEFT）方法如LoRA忽略MoE路由机制，导致资源利用不佳。本文提出EPnG，一种自适应剪枝与生长框架，依据路由器门控概率评估专家重要性，重新分配LoRA容量。该方法剪除低利用率专家，通过正交初始化实现高重要性专家的秩增长，同时保持固定参数预算。在OLMoE和Qwen1.5-MoE上，EPnG在相同预算下持续优于LoRA，性能媲美全量微调，仅需更新0.55%–0.72%参数（相比全量微调减少140x–180x），验证了将PEFT与MoE路由对齐可带来更高效、可扩展的微调策略。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation. Existing parameter-efficient fine-tuning (PEFT) methods such as LoRA ignore MoE routing dynamics, leading to suboptimal resource use. We propose EPnG, an adaptive prune-and-grow framework that reallocates LoRA capacity based on expert importance derived from router gate probabilities. EPnG prunes under-utilized experts and expands high-importance experts via rank growth with orthogonal initialization, while maintaining a fixed parameter budget. Across OLMoE and Qwen1.5-MoE, EPnG consistently outperforms LoRA under the same budget and achieves performance comparable to full fine-tuning while updating only 0.55%-0.72% of parameters (up to 140x-180x fewer). These results demonstrate that aligning PEFT with MoE routing yields a more effective and scalable fine-tuning strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。