arXiv:2606.31174cs.AI2026-06

评测大模型当团队主管的协调能力,发现权限管理是关键瓶颈。

ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents

论文配图:ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents
图 1 · 摘自论文原文
  • 设计新基准测试,让主模型指挥多个子代理完成复杂任务。
  • 发现权限分配准确率普遍低于50%,远低于感知能力限制。
  • 适合关注多智能体协作与系统架构的开发者和研究者。

生产级大语言模型代理正从独立解决问题转向担任管理者角色:主模型创建专业子代理,委派任务,并通过动态工作流协调其并行异步返回。但单个模型是否真能有效管理团队仍缺乏衡量标准——现有基准仅评估单一模型的任务解决能力或固定多代理系统的涌现行为,无法单独检验主模型的管理能力。我们提出ClawArena-Team,一个包含41个多轮、多模态、多目录场景的基准,覆盖258次评估轮次和72次阶段更新,专门测量这一管理能力。主模型被严格限制:仅原生感知文本,且只能访问部分工作区。它调用一组固定本地部署的子代理池,评分差异反映的是管理技巧而非原始能力。所有评分基于执行结果,无LLM评判:综合得分(子代理管理分,SMS)将任务正确性与最小权限、模态路由因子相乘。在十二种专有、社区托管及自托管模型上实验表明:管理瓶颈在于权限授予,而非感知能力(无模型工作区权限精度超过50%);成本与管理质量解耦(API成本跨度超100倍,总分跨度不足4倍,最便宜开源模型位于帕累托前沿);多数排行榜分数集中于9.9分区间,而编排行为差异超过一个数量级。代码已公开于https://github.com/aiming-lab/ClawArena。

原文摘要 · Abstract (English)

Production large language-model (LLM) agents are increasingly deployed not as lone problem-solvers but as managers: a main model creates specialized subagents, delegates work, and orchestrates their parallel, asynchronous returns through dynamic workflows. Whether one model can actually run such a team is largely unmeasured: existing benchmarks score a policy's own task-solving or a fixed multi-agent system's emergent behavior, but none isolate the management ability of the single LLM acting as leader. We introduce ClawArena-Team, a benchmark of 41 multi-turn, multimodal, multi-directory scenarios spanning 258 evaluation rounds and 72 staged updates that measures this management ability. The main agent is deliberately constrained: it natively perceives only text and directly accesses only part of the workspace. It commands a fixed, locally served subagent pool, so score differences reflect management skill, not raw capability. All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor. Across twelve proprietary, community-hosted, and self-hosted models, experiments show that the management bottleneck is privilege granting rather than perception (no model exceeds 50% workspace-permission precision); that cost and management quality are decoupled (API cost spans over 100 times while the overall score spans under 4 times, with the cheapest open models on the Pareto frontier); and that most leaderboard scores cluster within a 9.9-point band while orchestration behaviors diverge by more than an order of magnitude. Code is available at https://github.com/aiming-lab/ClawArena.

多智能体任务调度评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。