用编排轨迹研究大模型多智能体的强化学习，解决协作中的决策与奖励设计问题。
Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces
- 通过编排轨迹建模多智能体协作过程，捕捉任务生成、分配、通信等动态
- 发现停顿决策缺乏显式强化学习方法，奖励设计覆盖并行效率、拆分正确性等八类
- 开源84篇论文库和可复现的轨迹数据，助力学术与工业对比研究
随着大语言模型（LLM）智能体从孤立工具使用者演变为协同团队，强化学习（RL）需优化的不仅是个体行为，还包括任务的创建、委派、沟通、聚合与终止。本文通过编排轨迹（orchestration traces）研究基于LLM的多智能体系统中的强化学习：这些时间交互图包含子智能体生成、委派、通信、工具使用、返回、聚合及停止等事件。我们识别出三个技术维度：第一，奖励设计涵盖八类，包括促进并行加速、拆分正确性和聚合质量的编排奖励；第二，奖励与信用信号作用于从词元到团队的八个单元，其中消息级反事实信用仍极度稀缺；第三，编排学习可分解为五个子决策：何时生成、委派给谁、如何沟通、如何聚合、何时停止。截至2026年5月4日，在我们整理的文献池中，尚未发现针对停止决策的显式强化学习训练方法。我们将学术方法与来自Kimi Agent Swarm、OpenAI Codex、Anthropic Claude Code的工业证据相连接，揭示公开部署规模与学术评估体系之间的差距，并非独立验证工业训练轨迹。我们发布项目资源（https://github.com/xxzcc/awesome-llm-mas-rl），包含84条标记论文、32条排除记录、语料统计脚本及最小化可重放编排轨迹的JSON模式。
原文摘要 · Abstract (English)
As large language model (LLM) agents evolve from isolated tool users into coordinated teams, reinforcement learning (RL) must optimize not only individual actions but also how work is spawned, delegated, communicated, aggregated, and stopped. This paper studies RL for LLM-based multi-agent systems through orchestration traces: temporal interaction graphs whose events include sub-agent spawning, delegation, communication, tool use, return, aggregation, and stopping decisions. Using this lens, we identify three technical axes. First, reward design spans eight families, including orchestration rewards for parallelism speedup, split correctness, and aggregation quality. Second, reward and credit signals attach to eight credit- or signal-bearing units from token to team; explicit counterfactual message-level credit remains especially sparse in our curated pool. Third, orchestration learning decomposes into five sub-decisions: when to spawn, whom to delegate to, how to communicate, how to aggregate, and when to stop. In our curated pool as of May 4, 2026, we found no explicit RL training method for the stopping decision. We connect academic methods to public industrial evidence from Kimi Agent Swarm, OpenAI Codex, and Anthropic Claude Code. The resulting scale gap is a gap between publicly reported deployment envelopes and open academic evaluation regimes, not independent verification of industrial training traces. We release the artifact at https://github.com/xxzcc/awesome-llm-mas-rl, including an 84-entry tagged paper pool, a 32-record exclusion log, scripted corpus statistics, and a minimal JSON schema for replayable orchestration traces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。