构建首个通用终端智能体评估基准，覆盖真实办公与科研任务。
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
- 设计120个真实终端任务，涵盖文档、邮件、网页查询等日常操作。
- 顶尖智能体表现仅65.8%，显示通用终端能力仍有巨大提升空间。
- 专为非编程场景设计，适合评估跨领域数字助手的实用性。
随着大语言模型和工具调用框架的发展，终端智能体已能完成超出编码范围的通用计算机操作任务。然而，现有评估基准存在不足：通用型基准多面向图形界面，而终端类基准则偏重编程工作流。本文提出TUA-Bench，一个面向通用终端使用智能体（TUAs）的基准测试集。该基准包含120个真实世界任务，分为五大类，涵盖文档编辑、邮件管理、实时网络信息获取等日常数字活动，以及与博士级专家共同设计的科学与工程工作流，需使用专业软件。所有任务均手工设计，运行于真实终端环境，采用确定性部署脚本与执行评分协议。实验发现，表现最佳的智能体Claude Code搭配Claude Opus 4.8（最大推理资源）仅达65.8%整体性能，且在不同任务类别间差距显著。TUA-Bench旨在推动从专用助手向可信赖的通用终端智能体演进。
原文摘要 · Abstract (English)
As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。