提出新方法统一多智能体语言模型的协同决策与优化。
SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs
- 以最小输出集为单位进行联合优化，而非单个响应
- 在数学推理和多轮搜索任务中实现最优平均性能
- 支持固定、混合与动态路由工作流的统一训练
多智能体大语言模型通过多个策略在共享环境中协作解决复杂任务。然而，现有强化学习方法通常独立优化每个响应或轨迹，即使多个输出共同导致一次状态转移。这使得更新单元与系统实际执行的动作不一致。为此，我们提出SRPO（Setwise Relative Policy Optimization），将触发一次状态转移的最小输出集合视为一个联合动作。具体而言，SRPO将成员对数比合并为归一化集合比，分配单一相对优势并进行一次裁剪。该框架将分工与协同进化统一为不同集合大小的动作形式。在数学推理与多轮搜索任务上的实验表明，该方法可在四个模型规模下，统一支持固定、混合与动态路由工作流，且在报告结果中取得最强宏平均表现。优化诊断进一步刻画了其在不同事件减少与集合大小下的稳定性。
原文摘要 · Abstract (English)
Multi-agent large language models solve complex tasks by coordinating several policies in a shared environment. However, existing reinforcement learning methods usually optimize each response or trajectory separately, even when several outputs jointly cause one state transition. Consequently, the update unit differs from the action executed by the system. To address this problem, we propose SRPO (Setwise Relative Policy Optimization), which treats the active set the minimal set of outputs consumed by one transition, as one multi-agent action. Specifically, SRPO combines member log-ratios into one cardinality-normalized set ratio, assigns one relative advantage, and clips the set once. This formulation unifies division of labor and joint co-evolution as actions with different set sizes. Experiments on mathematical reasoning and multi-turn search demonstrate one training interface for fixed, mixed, and dynamically routed workflows across four model scales, with the strongest macro-average results among the reported comparisons. Optimization diagnostics further characterize its stability under different event reductions and set sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。