为计算机操作智能体设计组件级故障诊断基准，精准定位界面交互短板。
ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents
- 构建97类通用界面组件的2910个可验证任务，覆盖主流组件库。
- 发现同一模型在不同观察/动作空间下成功率相差超30%，最高差距达83.1%到48.9%。
- 提供可扩展的失败分析流水线，适合评估和优化智能体在真实网页中的交互能力。
当前计算机使用智能体的评估分为长流程工作流测试与原子级GUI对齐测试，中间层——即短时但丰富的组件级交互（如切换按钮组）缺乏有效评测工具。本文提出ComponentBench，一个面向现代网页界面的组件级评估基准与诊断流水线。该基准基于97类标准化UI组件，构建了2910个程序化验证的任务，涵盖广泛使用的组件库，并配有清洗后的真人参考轨迹，支持对任务成功率与交互效率的双重评估。除任务集外，我们还引入可扩展的审计管道，用于评估实现后的结构难度并生成跨任务与组件族的结构化失败分析。在四种观察与动作空间下，对七种模型（GPT-5.4、Gemini 3 Flash、GPT-5.4 mini、GPT-5 mini、Gemini 3.1 Flash-Lite、Qwen3-VL-235B、UI-TARS-1.5-7B）进行评估，结果表明设计选择显著影响性能：在同一统一框架下，仅改变观察与动作空间，GPT-5 mini的成功率从83.1%降至48.9%；即使最快配置也比匹配的人类参考轨迹慢3.7倍，且对人类而言简单的空间操作仍对当前智能体构成挑战。
原文摘要 · Abstract (English)
Current evaluation of computer-use agents is split between long-horizon workflow benchmarks and atomic GUI-grounding tests. This leaves an under-instrumented middle layer: realistic component-centered interactions (e.g., toggle a button set) that are short enough to diagnose and rich enough to capture the burdens of modern interfaces. We present ComponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs. ComponentBench is organized around a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks across widely used component libraries, paired with cleaned human reference trajectories that enable evaluation of both task success and interaction efficiency. Beyond task collection, we introduce a scalable pipeline for auditing realized structural difficulty after implementation and synthesizing structured failure analyses across tasks and component families. Evaluating seven models -- GPT-5.4, Gemini 3 Flash, GPT-5.4 mini, GPT-5 mini, Gemini 3.1 Flash-Lite, Qwen3-VL-235B, and UI-TARS-1.5-7B -- across four observation and action spaces, we show that these design choices critically impact performance. Within a single shared harness, changing only the observation and action space shifts task success by more than 30% for the same model: GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control. Moreover, even the fastest configuration takes 3.7x as long as the matched human reference, and spatial manipulations that are trivial for humans continue to challenge current agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。