测试大模型能否真实交付客服代理,揭示其在复杂环境下的实战短板。
$\tau^\tau$-Bench: An Environment for End-To-End, Realistic Agent Construction

- 以真实业务场景为起点,要求模型从记录、需求、接口等条件中自建完整客服代理。
- 最强模型仅通过23.9%的测试场景,远低于专家参考线82.2%。
- 暴露模型缺乏深度理解、沟通不足、架构试错少等实际开发痛点,适合研究智能工程能力者关注。
大语言模型代理正快速进入生产环境,用于客户服务、纠纷仲裁和内部系统运维。如今,构建这些代理的任务越来越多由编码代理承担,但现有基准难以评估其在真实客户场景中的交付能力。我们提出 τ^τ- bench(读作 hyper-tau-bench),一个将代理构建作为核心任务的基准。开发者代理需基于企业实际保存的记录、客户提出的需求、需调用的生产API、可继承的代码库,以及服务成本与模型使用的限制,从零构建完整的客户服务代理,并通过部署到预留的模拟用户上进行评分。在涵盖四个领域的53项任务中,表现最佳的配置(Claude Opus 5 + Claude Code)仅成功通过23.9%的评估场景,而专家编写的标准参考线可达82.2%。失败模式与人类开发者常见问题一致:模型倾向于提出浅层问题而非深入理解记录,几乎不与客户沟通,且对代理架构和资源消耗的探索不足,直接部署首个可运行的设计。我们旨在使协作式代理构建成为可量化的编码代理目标。
原文摘要 · Abstract (English)
LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement. We introduce $\tau^\tau$-bench (pronounced hyper-tau-bench), a benchmark that makes agent construction the task. A developer agent is given the records a business actually keeps, a client who holds requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models: the same starting point a real engagement provides. From these it must deliver a complete customer-service agent, scored by deploying that agent against held-out simulated users. Across 53 tasks spanning four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations. Meanwhile, an expert-authored reference ceiling scores 82.2%. The failures mirror ones human agent developers see: models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs. We aim for $\tau^\tau$-bench to turn the work of cooperative agent building into a measurable target for coding agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。