让多智能体对话更高效：用奖励模型判断中间消息价值
MASPRM: Multi-Agent System Process Reward Model
- 用奖励模型评估多智能体间消息序列，指导推理搜索
- 70亿模型在四基准上比基线最高提14.5分，且提升排名质量
- 无需人工标注每步，适合需要高效推理的AI系统
多智能体系统推理时，若无法识别哪个智能体的中间消息推动了进展，就会浪费计算资源。本文提出多智能体系统过程奖励模型（MASPRM），对智能体间消息传递序列进行打分，作为步级束搜索（SBS）和蒙特卡洛树搜索（MCTS）的推理控制器。MASPRM仅基于终端结果奖励训练，无需人类提供的步骤级标注。在GSM8K、MATH、MMLU和LogiQA四个数据集上评估，与同规模基线相比，在15亿参数下提升2.0至3.0分，70亿参数下提升4.1至14.5分；70亿模型下，相比策略似然还额外获得平均13.4分提升（使用MCTS时）。同时，该方法显著改善排序质量，使命中率从Hit@1到Hit@5的差距缩小最多达10.3个百分点，尤其在依赖中间决策的逐步搜索中收益最大。代码已开源。
原文摘要 · Abstract (English)
Inference-time search over multi-agent systems (MAS) wastes compute when it cannot identify which agent's intermediate message advanced progress. We present the Multi-Agent System Process Reward Model (MASPRM), which scores routed transcripts (ordered sequences of messages between agents) and acts as an inference controller for step-level beam search (SBS) and Monte Carlo Tree Search (MCTS). MASPRM is trained from multi-agent MCTS rollouts labeled only with terminal outcome rewards, without human step-level annotations. We evaluate on GSM8K, MATH, MMLU, and LogiQA. Under matched scorer size and comparable MCTS budget, MASPRM exceeds a size-matched ORM by $+2.0$ to $+3.0$ points at 1.5B and $+4.1$ to $+14.5$ at 7B across all four benchmarks, with additional scorer-scaling gains over policy likelihood at 7B (avg $+13.4$ under MCTS). MASPRM also improves ranking quality, reducing Hit@1 to Hit@5 gaps by up to $10.3$ points, with the largest gains under stepwise search that uses intermediate decisions. Code: https://github.com/milad1378yz/MASPRM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。