通过角色分工与混合部署，显著降低大模型代理团队成本并提升准确率。
Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams
- 按角色分配不同模型，支持API、自托管和混合部署的灵活组合。
- 同成本下准确率最高提升44%，或用12倍更低成本达到最强性能。
- 适用于多角色代理系统，可动态评估新领域与团队结构。
大语言模型代理正以多角色团队形式部署，任务在规划者、执行者、验证者等角色间分配。此时成本与准确率不再取决于单一模型，而受角色分配及部署方式（API、自托管、混合）影响。现有基准通常评估固定模型或配置，难以指导最优部署。我们提出AgentCARD，一个面向角色与部署模式的角色感知评估套件，包含角色分解评测框架、统一成本模型、帕累托前沿分析及基于Shapley值的瓶颈诊断。评估显示，异构团队始终位于成本-准确率前沿：相比同成本下的同质团队，准确率最高提升44%；或在保持最强同质团队性能的同时，将每任务成本降低至1/12。最佳角色分配具有领域依赖性：部分领域受规划者瓶颈制约，另一些则受限于执行者能力。此外，AgentCARD扩展至含验证等多角色流程，并支持随新领域与团队结构演进的持续评估。代码已开源：https://github.com/Auto-CAP/AgentCAP
原文摘要 · Abstract (English)
LLM agents are increasingly deployed as multi-role teams, where tasks are divided across specialized roles such as planner, executor, and verifier. In these systems, cost and accuracy are no longer properties of a single model: they depend on which model fills each role and where it is hosted, including API, self-hosted, and hybrid deployment. Existing agentic benchmarks typically evaluate fixed models or fixed agent configurations, and therefore offer limited guidance for cost-accuracy-optimal deployment. We introduce AgentCARD, a role-aware benchmark suite for evaluating LLM agent teams across role assignment and deployment mode. AgentCARD combines a role-decomposed evaluation harness, a unified API/self-hosted cost model, Pareto-frontier analysis, and a Shapley-based diagnostic for identifying role bottlenecks. Our evaluation shows that heterogeneous teams consistently occupy the cost-accuracy frontier. They improve accuracy by up to $44\%$ over cost-equivalent homogeneous teams, or match the strongest homogeneous team at up to $12\times$ lower per-task cost through hybrid deployment. We further find that the best role assignment is domain-dependent: some domains are planner-bottlenecked, while others are executor-bottlenecked. Finally, AgentCARD extends beyond planner--executor teams to workflows with additional roles such as verification, and supports continual evaluation as new domains and team structures emerge. Our code is released at: https://github.com/Auto-CAP/AgentCAP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。