只微调热门专家,训练效率提升50%且精度几乎不变
MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
- 根据路由频率筛选每层最活跃的25%专家进行微调
- 仅微调热门专家时,精度与全量微调相差不超过±1个百分点
- 适合追求高效微调的模型优化者,尤其适用于大模型
标准的LoRA微调对MoE模型的所有专家应用适配器,但我们的分析显示,每层专家的路由分布极不均衡:少数专家处理了大部分输入,而许多专家几乎未被激活(冷专家)。我们提出MoE-Sieve,一种基于路由引导的简单微调框架,并系统性地分析了不同架构和任务下的专家路由行为。方法为:在小规模校准集上统计路由频次,每层选取前k个最常被路由的专家,仅对这些专家应用LoRA。在两个架构不同的MoE模型及三个多样化任务上,仅微调每层前25%路由频率最高的专家,性能仍可媲美全量微调,平均误差在±1个百分点内。该方法使可训练参数减少70-73%,适配器检查点大小缩减71-73%,训练时间最多缩短50%。我们还发现专家数量与种子间方差之间存在非单调关系,支持‘微调冷专家会引入梯度噪声而不提升准确率’的假设。消融实验表明,在相同预算下随机选择专家性能低约2.5个百分点,说明路由信号至关重要;而逐层贪婪预算分配也未优于均匀的top-k策略。
原文摘要 · Abstract (English)
Standard LoRA fine-tuning of Mixture-of-Experts (MoE) models applies adapters to every expert, yet our profiling shows that per-layer expert routing is highly skewed: a small subset of experts handles most tokens in each layer, while many others are rarely activated ("cold"). We propose MoE-Sieve, a simple routing-guided framework for LoRA fine-tuning, and pair it with a systematic profiling study of expert routing across architectures and tasks. The method is simple: profile routing counts on a small calibration set, select the top-k most-routed experts per layer, and apply LoRA only to those experts. Across two architecturally distinct MoE models and three diverse tasks, tuning only the top 25% routed experts per layer remains competitive with full LoRA, with mean differences within +/-1 percentage point across all conditions. This reduces LoRA trainable parameters by 70-73%, adapter checkpoint size by 71-73%, and wall-clock training time by up to 50%. We also observe a non-monotonic relationship between expert count and seed-to-seed variance, consistent with the hypothesis that adapting cold experts can introduce gradient noise without improving accuracy. Further ablations show that random expert selection at matched budget is about 2.5 percentage points worse, indicating that the routing signal matters, while greedy per-layer budget optimization does not improve over uniform top-k.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。