无需人工标注，用中间结果训练多智能体协调奖励模型。
Reward Modeling for Multi-Agent Orchestration
- 基于多智能体执行的中间产物构建胜负对，训练奖励模型。
- 训练效率提升10倍，测试时扩展准确率最高提升8%。
- 适用于数学推理、网页问答等多领域，适合研究者复现。
基于大语言模型的多智能体系统需要有效协调以整合专用智能体，但其协调器训练受限于监督信号不足和计算成本高。我们提出协同奖励建模（OrchRM），一种无需人工标注的自监督框架，通过多智能体执行过程中的中间产物构造胜负对，用于Bradley-Terry奖励模型训练。与依赖昂贵子智能体回滚的传统方法不同，OrchRM直接在协调层级运作，实现高效且高性能的奖励引导式协调器训练及多智能体系统测试时扩展。实验表明，OrchRM在令牌使用上效率提升达10倍，测试时扩展准确率最高提升8%，并在数学推理、基于网络的问答和多跳推理等多个领域表现出一致性能增益，验证了协调层级奖励建模作为可扩展、鲁棒的多智能体协调方向的潜力。代码将开源至 https://github.com/Wang-ML-Lab/OrchRM。
原文摘要 · Abstract (English)
Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents, yet training such orchestrators is hindered by limited supervision and high computational cost. We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations. OrchRM leverages intermediate artifacts from multi-agent executions to construct win-lose pairs for Bradley-Terry reward model training. Unlike existing MAS test-time scaling and orchestrator training frameworks that rely on costly sub-agent rollouts, OrchRM operates directly at the orchestration level, enabling efficient and high-performing reward-guided orchestrator training and MAS test-time scaling. OrchRM improves training efficiency by up to 10x in token usage while improving MAS test-time scaling performance by up to 8% in accuracy. These gains consistently transfer across multiple domains, including mathematical reasoning, web-based question answering, and multi-hop reasoning, demonstrating orchestration-level reward modeling as a scalable direction for robust multi-agent orchestration. Code will be available at https://github.com/Wang-ML-Lab/OrchRM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。