用反事实强化学习让大模型自动设计更高效协作的多智能体系统。
LEMON: Learning Executable Multi-Agent Orchestration via Counterfactual Reinforcement Learning
- 基于反事实信号优化角色、容量与依赖关系的协同设计。
- 在6个基准上达到当前最优性能，显著提升任务解决效率。
- 适合需要自动化多智能体协作设计的研究者与工程师。
大型语言模型（LLMs）已成为多智能体系统的重要基础，但其效果高度依赖于编排设计。角色设定、资源配置和依赖关系共同影响解决方案质量与执行效率。现有方法虽能部分自动化设计过程，但通常局部或顺序优化，且依赖执行反馈，难以精准分配局部决策的奖励。我们提出 LEMON（Learning Executable Multi-Agent Orchestration via Counterfactual Reinforcement Learning），一种基于 LLM 的编排器，可生成可执行的编排规范，整合任务特定角色、定制职责、容量层级与依赖结构。训练时，通过局部反事实信号修改角色、容量或依赖字段，并仅对修改部分应用奖励对比。在 MMLU、GSM8K、AQuA、MultiArith、SVAMP、HumanEval 六个推理与编码基准上的实验表明，LEMON 在所评估的多智能体编排方法中表现最佳。代码已公开于 https://anonymous.4open.science/r/LEMON-B23C。
原文摘要 · Abstract (English)
Large language models (LLMs) have become a strong foundation for multi-agent systems, but their effectiveness depends heavily on orchestration design. Across different tasks, role design, capacity assignment, and dependency construction jointly affect both solution quality and execution efficiency. Existing approaches automate parts of this design process, yet they often optimize these decisions partially or sequentially, and rely on execution-level feedback that provides limited credit assignment for local orchestration decisions. We propose LEMON (\textbf{L}earning \textbf{E}xecutable \textbf{M}ulti-agent \textbf{O}rchestratio\textbf{N} via Counterfactual Reinforcement Learning), an LLM-based orchestrator that generates an executable orchestration specification. The specification integrates task-specific roles, customized duties, capacity levels, and dependency structure into a single deployable system. To train the orchestrator, we augment the orchestration-level GRPO objective with a localized counterfactual signal that edits role, capacity, or dependency fields and applies the resulting reward contrast only to the edited spans. Experiments on six reasoning and coding benchmarks, including MMLU, GSM8K, AQuA, MultiArith, SVAMP, and HumanEval, show that LEMON achieves state-of-the-art performance among the evaluated multi-agent orchestration methods. Our code is available at https://anonymous.4open.science/r/LEMON-B23C.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。