测试大模型在信息不全时的协作能力，发现推理强未必合作好。
CRAFT: Grounded Multi-Agent Coordination Under Partial Information
- 设计多智能体任务：各智能体仅知部分信息，需用自然语言协作建3D结构。
- 发现小模型常胜过大模型，单独沟通好不代表能成功协作。
- 提供诊断框架，可拆解空间定位、信念建模和语用沟通三类失败。
我们提出CRAFT，一个用于评估大语言模型在严格部分信息下实用通信能力的多智能体基准。在此设定中，多个拥有互补但不完整视角的智能体需通过自然语言协调，构建单个智能体无法完全观测的共享3D结构。我们将该问题形式化为多发送方有界语用说话人问题，并提供一个诊断框架，将失败分解为空间接地、信念建模和语用通信错误，包括对前沿与开源模型的行为失败类型分类。在涵盖8个开源模型和7个前沿模型（含推理模型）的多样化测试中，我们发现更强的推理能力并不保证更好的协作：较小的开源模型时常表现匹配或优于前沿系统，且个体沟通能力提升也不确保协作成功。这些结果表明，当前语言模型在多智能体协作方面仍面临根本性挑战。代码已公开于https://github.com/csu-signal/CRAFT。
原文摘要 · Abstract (English)
We introduce CRAFT, a multi-agent benchmark for evaluating pragmatic communication in large language models under strict partial information. In this setting, multiple agents with complementary but incomplete views must coordinate through natural language to construct a shared 3D structure that no single agent can fully observe. We formalize this problem as a multi-sender Bounded Pragmatic Speaker problem and provide a diagnostic framework that decomposes failures into spatial grounding, belief modeling and pragmatic communication errors, including a taxonomy of behavioral failure profiles in both frontier and open-weight models. Across a diverse set of models, including 8 open-weight and 7 frontier including reasoning models, we find that stronger reasoning ability does not reliably translate to better coordination: smaller open-weight models often match or outperform frontier systems, and improved individual communication does not guarantee successful collaboration. These results suggest that multi-agent coordination remains a fundamentally unsolved challenge for current language models. Our code can be found at https://github.com/csu-signal/CRAFT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。