测试大模型智能体在真实工作场景中的自动化能力，发现仅30%任务可自主完成。
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
- 构建模拟软件公司环境，评估智能体网页浏览、编程、协作等真实工作能力。
- 最先进智能体仅能自主完成30%的任务，复杂长周期任务仍难胜任。
- 适合关注AI办公自动化与未来劳动力影响的研究者和企业决策者。
我们日常工作中大量依赖计算机和互联网，而随着大语言模型（LLMs）的发展，能够与环境交互并产生影响的AI智能体也快速兴起。然而，这些智能体在加速或自主执行工作相关任务方面的表现如何？这一问题对产业采用AI及经济政策制定具有重要意义。为衡量此类智能体在真实职业任务上的进展，本文提出TheAgentCompany——一个可扩展的基准测试框架，用于评估智能体在类似数字员工的环境中（如网页浏览、编写代码、运行程序、与其他同事沟通）的表现。我们构建了一个包含内部网站和数据的自包含环境，模拟小型软件公司，并设计了多种典型工作任务。测试了基于封闭API和开源权重的语言模型驱动的基线智能体，结果显示最优秀的智能体可自主完成30%的任务。这揭示出一个复杂图景：在模拟真实职场的场景中，部分简单任务可实现自主解决，但更复杂的长期任务仍超出当前系统能力。代码、数据、环境和实验已公开于https://the-agent-company.com。
原文摘要 · Abstract (English)
We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at accelerating or even autonomously performing work-related tasks? The answer to this question has important implications both for industry looking to adopt AI into their workflows and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that the most competitive agent can complete 30% of tasks autonomously. This paints a nuanced picture on task automation with LM agents--in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems. We release code, data, environment, and experiments on https://the-agent-company.com.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。