arXiv:2604.18473cs.LG2026-04被引 1

让大模型分领域独立训练,再用轻量路由组合,更新快且不丢旧能力。

Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts

  • 分领域独立训练专家,用轻量路由融合,避免全模型重训
  • 7B模型在7个评测项上达49.1分,优于或持平全量重训基线
  • 适合需持续扩展功能且怕遗忘旧知识的工业级模型部署

扩展已完全后训练的语言模型以新增领域能力,受限于单一训练范式:从头重训成本高且扩展性差,持续训练又常导致已有能力退化。我们提出BAR(Branch-Adapt-Route),分别训练独立的领域专家,每个专家通过自身中段训练、监督微调和强化学习流程完成,再通过混合专家架构与轻量路由进行组合。相比需混合所有领域并全量重处理的重训方法(成本随更新次数呈二次增长),BAR可独立更新单个专家,成本仅线性增长,且不损害已有领域性能。在70亿参数规模下,包含数学、代码、工具使用和安全四个领域专家的BAR,在7个评估类别上平均得分49.1,优于或达到重训基线(无中段训练为47.8,有中段训练为50.5)。进一步表明,模块化训练具有结构优势:通过隔离各领域,避免晚期强化学习对前期训练能力造成的灾难性遗忘,同时显著降低更新或新增领域的成本与复杂度。结果表明,解耦的专家式训练是扩展语言模型的一种可扩展替代方案。

原文摘要 · Abstract (English)

Extending a fully post-trained language model with new domain capabilities is fundamentally limited by monolithic training paradigms: retraining from scratch is expensive and scales poorly, while continued training often degrades existing capabilities. We present BAR (Branch-Adapt-Route), which trains independent domain experts, each through its own mid-training, supervised finetuning, and reinforcement learning pipeline, and composes them via a Mixture-of-Experts architecture with lightweight router training. Unlike retraining approaches that mix all domains and require full reprocessing for any update (with cost scaling quadratically), BAR enables updating individual experts independently with linear cost scaling and no degradation to existing domains. At the 7B scale, with experts for math, code, tool use, and safety, BAR achieves an overall score of 49.1 (averaged across 7 evaluation categories), matching or exceeding re-training baselines (47.8 without mid-training, 50.5 with). We further show that modular training provides a structural advantage: by isolating each domain, it avoids the catastrophic forgetting that occurs when late-stage RL degrades capabilities from earlier training stages, while significantly reducing the cost and complexity of updating or adding a domain. Together, these results suggest that decoupled, expert-based training is a scalable alternative to monolithic retraining for extending language models.

大模型扩展MoE后训练模块化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。