模型合并效果媲美联合训练，关键在于任务向量近正交。
When Model Merging Rivals Joint Multi-Task Reinforcement Learning: A Task-Vector Geometry Analysis
- 通过任务向量几何分析发现，不同任务的参数方向近乎正交。
- 合并后的模型在目标任务完成率上与联合训练无统计差异。
- 适合关注模型融合机制与多任务学习的算法研究者阅读。
模型合并被视作联合多任务训练的替代方案，但在强化学习场景中，这一替代从未真正与原基线进行对比：独立发布的智能体之所以合并，正是因为无法获得联合模型。本文首次构建了这一缺失的对比。在AppWorld基准上，使用LOOP训练难度1和难度2的Qwen3-8B专用模型，并采用TIES、RAM+方法进行合并，将其结果与同一数据上联合训练的模型进行对比。在任务目标完成率上，合并模型表现与联合训练相当，且所有合并变体均无统计差异。为解释为何合并方法不影响结果，我们测量了专用模型的任务向量几何结构，其任务采样噪声极低：尽管支持重叠率达约65%，但任务向量间余弦相似度仅为0.06–0.10，呈现近正交性；存在一个随训练增长的小共享方向，经随机初始化下界和同次实验上界校准，确认其反映真实学习过程，而非低秩参数化导致。由于方向与支持解耦，基于支持或符号的合并（RAM、TIES）退化为近似均匀平均。代码与全部统计数据均已开源。
原文摘要 · Abstract (English)
Model merging is promoted as a substitute for joint multi-task training, yet in the reinforcement-learning setting this substitution is essentially never tested against the baseline it claims to replace: methods merge independently released agents precisely because a joint model is unavailable. We build the missing comparison. Training difficulty-1 and difficulty-2 Qwen3-8B specialists on the AppWorld agent benchmark with LOOP, we merge them (TIES, RAM+) and pit the result against a jointly trained model on the same data. On task-goal completion, merging matches joint RL -- and every merge variant is statistically indistinguishable. To explain why merge method does not matter here, we measure the geometry of the specialists' task vectors, which carries no task-sampling noise: they are near-orthogonal (cosine 0.06 - 0.10) despite ~65% support overlap, a small, shared direction that grows over training and that we calibrate against a random-init floor and a same-run ceiling to confirm it reflects learning, not the low-rank parameterization. Because direction and support are decoupled, support and sign-based merging (RAM, TIES) collapse to near-uniform averaging. We release all code and statistics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。