arXiv:2608.07890cs.LGcs.AI2026-08

轻量微调中路由器敏感度可精准识别可剪枝专家,大幅降低MoE模型开销。

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

论文配图:Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
图 1 · 摘自论文原文
  • 用轻量适配器微调,按路由器变化量排序专家并一次性剪枝。
  • 剪掉一半专家后仍保持27.54%准确率,内存降49%,延迟减37%。
  • 方法适用于多任务场景,且可迁移至其他MoE模型,适合部署优化。

Mixture-of-Experts (MoE) 模型将总参数量与每令牌计算量解耦,但部署时仍需存储所有专家。近期理论表明,剪掉微调中路由器范数变化最小的专家可保持精度,但假设为全参数微调。本文检验轻量级适应是否能恢复该信号:仅用参数高效的适配器进行短时微调,根据诱导出的ℓ₂路由器变化量对专家排序,并单次剪枝变化最小者。在 Mixtral-8×7B-Instruct(MMLU-Pro 44.83%)上,仅微调路由器的 LoRA 仅训练 0.002% 参数,且在相同秩下优于全模块 LoRA,半数专家被剪后准确率达 27.54%(对比 24.42%);当适应扩展至注意力和专家权重时,信号质量下降。准确率随 LoRA 秩单调提升,最高达 28.76%。IA3(冻结路由器)表现与直接微调路由器相当,而无约束加性适配器破坏信号。基于路由器引导的 MMLU-Pro 准确率呈准线性衰减,最大压缩下仍为基于幅度或随机剪枝的 1.8 倍,内存减少 49%,每令牌延迟降低 37%。在 25% 压缩率下,性能媲美使用完整激活统计的方法。该准则亦成功迁移至数学任务微调的 Qwen1.5-MoE,剪掉一半专家后平均准确率仍保持 49.7%(11 个基准),随机剪枝则降至个位数。因此,轻量微调下的路由器敏感度使可证明动机的专家剪枝在大规模应用中成为可能。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert. Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can preserve accuracy, but assumes full fine-tuning. We test whether lightweight adaptation can recover this signal. We briefly fine-tune with a parameter-efficient adapter, rank experts by the induced $\ell_2$ router change, and prune the least-changed experts in one shot. On Mixtral-8$\times$7B-Instruct (44.83% MMLU-Pro), router-only LoRA trains 0.002% of parameters and outperforms all-module LoRA at matched rank with half the experts removed (27.54% vs. 24.42%); signal quality declines as adaptation spreads to attention and expert weights. Accuracy improves monotonically with LoRA rank, reaching 28.76%. IA3, which leaves router weights frozen, matches direct router adaptation, whereas unconstrained additive adapters degrade the signal. Router-guided MMLU-Pro accuracy decays quasi-linearly rather than collapsing, remains nearly 1.8 times that of magnitude-based or random pruning at maximal compression, and reduces memory by 49% and per-token latency by 37%. At 25% compression, retention is competitive with methods using full activation statistics. The criterion also transfers to Qwen1.5-MoE fine-tuned for mathematics, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed while random pruning falls to single digits. Router sensitivity under lightweight fine-tuning therefore makes provably motivated expert pruning practical at scale.

MoE专家剪枝轻量微调模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。