arXiv:2601.13572cs.LG2026-01ACL被引 11

针对强化学习训练的智能体,提出新融合方法提升通用模型性能。

Behavior Knowledge Merge in Reinforced Agentic Models

  • 分离共享与任务特异性参数更新,针对性融合
  • 在多领域测试中超越专用智能体表现
  • 适合需要融合多个强化学习智能体的场景

强化学习(RL)在后训练阶段对具备专门推理行为的智能体至关重要。模型融合为将不同任务训练出的多个RL智能体整合为单一通用模型提供了实用机制。然而,现有融合方法针对监督微调(SFT)设计,难以保留RL训练智能体的任务特异性能力。根源在于RL与SFT间任务向量的不匹配:在线策略RL产生的任务向量高度稀疏且异质,而SFT式融合隐含假设向量稠密且全局可比。标准全局平均在此不匹配下会削弱非重叠的任务向量,导致关键行为参数更新被稀释。为此,我们提出强化智能体融合(RAM),一种专为RL训练智能体设计的分布感知融合框架。RAM解耦共享与任务特异性参数更新,对共享部分进行平均,同时选择性保留并重缩放特异性部分以对抗参数更新稀释。在多个智能体领域和模型架构上的实验表明,RAM不仅超越基线融合方法,还激发智能体间的协同潜力,在各自领域表现优于专用智能体。

原文摘要 · Abstract (English)

Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models. The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted. To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.

强化学习智能体融合模型合并

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。