通过强制角色分离评估智能体协作，揭示了传统评估的虚假通过率问题。
TeamBench: Evaluating Agent Coordination under Enforced Role Separation
- 用操作系统级权限隔离规划、执行、验证三角色，禁止越权操作。
- 无权限控制时通过率与有控制时无显著差异，但越权行为多出3.6倍。
- 适合研究多智能体协作机制或需真实分工评估的场景。
智能体系统常将任务拆分为多个角色，但这些角色通常仅通过提示词定义，而非由访问控制强制执行。缺乏强制约束时，团队通过率可能掩盖实际协作情况，或只是某个角色在代做其他角色的工作。本文提出 TeamBench，一个包含 851 个任务模板和 931 个种子实例的基准，用于在操作系统强制的角色分离下评估智能体协作。该基准将规范访问、工作区编辑和最终认证分别分配给 Planner、Executor 和 Verifier 三个角色，确保任一角色无法同时读取完整需求、修改工作区或认证最终答案。结果显示，仅靠提示词和沙箱强制的团队通过率无统计学差异，但提示词方式下验证者尝试修改执行者代码的案例多出 3.6 倍。验证者批准了 49% 在确定性评分器中失败的提交，且移除验证者后平均部分得分提升。团队价值具有条件性：当单个智能体表现不佳时团队受益，但当单个智能体已表现良好时反而受损。一项 40 次会话的人类实验表明，该基准能暴露通过率无法捕捉的交互模式：单独参与者直接完成任务，人机配对常演变为快速批准，而人类团队则花费更多精力协调跨角色信息缺失。
原文摘要 · Abstract (English)
Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls. Without enforcement, a team pass rate can mask whether agents actually coordinated or whether one role effectively did another role's work. We present TeamBench, a benchmark with 851 task templates and 931 seeded instances for evaluating agent coordination under operating system-enforced role separation. TeamBench separates specification access, workspace editing, and final certification across Planner, Executor, and Verifier roles, so that no role can read the full requirements, modify the workspace, and certify the final answer. Prompt-only and sandbox-enforced teams reach statistically indistinguishable pass rates, but prompt-only runs produce 3.6 times more cases where the verifier attempts to edit the executor's code. Verifiers approve 49% of submissions that fail the deterministic grader, and removing the verifier improves mean partial score in the ablation. Team value is also conditional. Teams benefit when single agents struggle, but hurt when single agents already perform well. A 40-session human study under the same role separation shows that our benchmark exposes interaction patterns that pass rate misses. Solo participants work through the task directly, human participants paired with agents often collapse into quick approval, and human teams spend more effort coordinating missing information across roles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。