arXiv:2606.03391cs.LGcs.AI2026-06

解决MoE模型合并时路由失效问题,无需训练即可提升性能

When Model Merging Breaks Routing: Training-Free Calibration for MoE

论文配图:When Model Merging Breaks Routing: Training-Free Calibration for MoE
图 1 · 摘自论文原文
  • 利用二阶曲率信息校准合并后的路由机制
  • 在数学推理和代码生成任务上显著提升性能
  • 适合需要高效整合多个专家模型的场景

模型合并已成为在不重新训练的情况下整合多个大语言模型能力的经济高效方法。然而,现有合并技术主要基于线性参数算术或优化,在应用于混合专家(MoE)架构时表现不佳。我们识别出一种关键故障模式——路由崩溃,即合并后的路由器无法将标记正确分发给合适的专家。路由崩溃源于非线性Softmax和离散Top-k路由机制对合并引起的参数扰动敏感,而预训练阶段施加的负载均衡约束进一步加剧了这种敏感性。由于微调后的专家具有不同专长,即使轻微的错误路由也会导致严重性能下降。为此,我们提出无需训练的赫森感知路由校准(HARC)框架,利用二阶曲率信息重新对齐合并后的路由器。该方法可导出闭式解,并通过无矩阵共轭梯度法高效求解。在数学推理和代码生成任务上的实验表明,HARC能有效缓解多种MoE合并基线中的路由崩溃问题,并带来显著性能提升。代码已开源:https://github.com/huangcb01/HARC。

原文摘要 · Abstract (English)

Model merging has emerged as a cost-effective approach for consolidating the capabilities of multiple LLMs without retraining. However, existing merging techniques, largely based on linear parameter arithmetic or optimization, struggle when applied to Mixture-of-Experts (MoE) architectures. We identify a critical failure mode in MoE merging, termed routing breakdown, in which the merged router fails to dispatch tokens to suitable experts. Routing breakdown stems from the sensitivity of the non-linear softmax and discrete Top-k routing mechanisms to parameter perturbations from merging, a sensitivity further amplified by load-balancing constraints imposed during MoE pretraining. Because fine-tuned experts exhibit distinct specializations, even modest misrouting can cause severe performance degradation. To address this issue, we propose Hessian-Aware Router Calibration (HARC), a training-free framework that leverages second-order curvature information to realign the merged router. This approach admits a closed-form solution that can be efficiently solved using a matrix-free conjugate gradient method. Experiments on mathematical reasoning and code generation tasks show that HARC effectively mitigates routing breakdown across diverse MoE merging baselines and leads to substantial performance improvements. Our code is available at https://github.com/huangcb01/HARC.

MoE模型合并路由校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。