arXiv:2608.27409cs.CL2026-08

对比三种模型融合方法,指导如何在多领域任务中高效整合强化学习能力。

Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms

论文配图:Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
图 1 · 摘自论文原文
  • 提出三种融合范式:合并任务向量、混合数据集、多教师在线蒸馏。
  • 不同方法在单一任务上表现差异达8.6分,跨领域关系可从任务向量几何中观察。
  • 推荐场景明确:合并适合已有专家且追求低成本,混合适合无专家时训练统一模型。

强化学习结合可验证奖励(RLVR)能提升大语言模型的特定能力,但覆盖多种能力通常需分别训练领域专家,再进行整合。本文归纳了三种融合范式:合并(Merge)复用专家任务向量,混合(Mix RL)复用其数据集,多教师在线蒸馏(MOPD)同时复用两者。由于这些方法长期被孤立研究,其对比关系与选择标准尚不清晰。本文在共享专家与数据的基础上,通过多模型规模与多领域基准测试对三者进行系统比较。尽管平均性能差异不超过1.4分,但在单个基准上最高相差8.6分,且领域间性能变化与任务向量几何中的跨域关系一致。训练动态显示:混合依赖领域比例,多教师蒸馏受教师上限约束,而合并将所有专家更新压缩为单一操作。三者均提升单样本准确率,但未显著改善解空间覆盖率,也未损害保留能力。由此提出实用建议:当已有专家且追求低成本融合时选合并;训练统一模型而无专家时,调整领域比例以优化跨域迁移则用混合;若需保留领域特异性优势而非超越教师或降低端到端成本,则优选多教师蒸馏。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artefacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.

强化学习模型融合多领域语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。