测试大模型多智能体协作能力,发现现有方法无法突破瓶颈。
GRPO Does Not Close the Multi-Agent Coordination Gap

- 用哲学家就餐问题测试多智能体协作,使用多种模型和规模
- 前沿闭源系统平均得分0.45-0.87,而14B开源模型仅0.13-0.35
- 训练方法缺陷是关键,需改进奖励设计与检查点策略
我们通过哲学家就餐问题这一清晰的测试基准,评估当前大型语言模型作为共享资源的多个智能体时的协作能力。在涵盖七种模型和三种哲学家数量的630个实验周期中,四种前沿闭源系统平均得分为0.45至0.87,Mistral-Small 24B得分为0.83至0.99,而Qwen3-14B仅为0.13至0.35。随后我们检验基于任务自身回放的组相对策略优化(GRPO)能否缩小差距,结果表明其无效:五位哲学家情况下每轮得分的Welch's t检验p值为0.66,Hedges' g为-0.11,十位和十五位哲学家也无统计显著变化。此外,8B和14B模型的训练奖励在第9步达峰值后下降,因此默认保存的第15步检查点反而劣于更早版本。所用四元奖励函数存在零动作下的退化解,导致DeepSeek-R1-Distill-Qwen-7B和Mistral-Small 24B在五位哲学家下均陷入零餐状态,平均得分分别为1.0和0.83。开放权重的14B模型在多智能体协作中的瓶颈并非训练算力,而是训练方法:避免奖励坍缩至零动作、不依赖最终步骤的检查点策略,以及跨问题规模的课程设计。
原文摘要 · Abstract (English)
We measure how well current large language models coordinate as multiple agents sharing a common resource, using the dining philosophers problem as a clean test bed. Across 630 episodes spanning seven models and three philosopher counts, four frontier closed-source systems reach mean reward 0.45 to 0.87 and Mistral-Small 24B reaches 0.83 to 0.99, while Qwen3-14B reaches 0.13 to 0.35. We then ask whether group relative policy optimization (GRPO) on rollouts from the task itself can close the gap and find that it cannot: a Welch's t-test on per-episode reward at five philosophers gives p = 0.66 and a Hedges' g of -0.11, with no statistically significant change at ten or fifteen philosophers either. Two further observations qualify the result. The training reward of both 8B and 14B runs peaked at step nine and then declined, so the default saved checkpoint at step 15 is strictly worse than several earlier ones. The four-term reward we use admits a degenerate maximum at zero actions, which DeepSeek-R1-Distill-Qwen-7B and Mistral-Small 24B at five philosophers both inhabit, with mean reward 1.0 and 0.83 respectively at zero meals. The bottleneck for an open-weight 14B model on multi-agent coordination is not training compute but training methodology: reward shaping that does not collapse to a no-action maximum, checkpoint discipline that does not depend on the final step, and curriculum across problem scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。