通过语义状态追踪，揭示网页智能体失败的深层原因。
Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking
- 构建带语义状态追踪的WebStep基准，自动记录交互过程
- 发现三款代理成功率达34%-37%但探索与执行表现迥异
- 可定位具体技能短板，指导针对性优化
网页智能体需完成长序列交互，但现有评估仅关注最终结果，忽略过程信息。本文提出WebStep基准，包含1800个任务实例，具备可控难度和自动语义状态追踪。每个网站同时提供确定性语义马尔可夫决策过程（MDP）与图形界面：智能体操作界面，环境在后台记录高层状态与转移，实现无需人工标注的细粒度分析。基于语义轨迹，我们发现过程指标揭示了结果评估无法捕捉的差异——三款智能体成功率均在34%-37%之间，但在探索覆盖范围与执行准确率上表现不同。按技能分解显示，同一网站内各智能体排名相反：例如在问答任务中，Claude CUA导航动作优于OpenAI CUA 30%，但检查动作却低6.7%。二分法分析进一步定位导致任务失败的关键错误，且该错误具有智能体特异性。随着任务难度上升，差异愈发显著：简单任务下成功率相近，复杂探索任务中则明显分化。本研究为网页智能体评估开辟新路径，提供精细、可行动的改进方向。
原文摘要 · Abstract (English)
Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and offering little guidance on improvement. In this work, we conduct a process-level analysis of web agents. We introduce WebStep, a benchmark of 1,800 task instances with controlled difficulty and automatic semantic state tracking. Each website exposes a deterministic semantic MDP alongside the GUI: the agent operates on the interface, while the environment records high-level states and transitions in the background, enabling fine-grained analysis without manual annotation. Based on the semantic trajectory, we first show that process metrics reveal differences invisible to outcome evaluation: three agents whose success rates cluster within 34-37% diverge in exploration reach versus execution accuracy. Then, decomposing by skill characterizes the nature of these differences, exposing opposite per-skill rankings hidden within the same website: e.g., on Q&A, Claude CUA outperforms OpenAI CUA by 30% on navigation actions yet underperforms it by 6.7% on inspection, pinpointing a concrete skill to improve even within a domain. Bifurcation analysis further localizes the decisive error that loses the task and shows that this error is agent-specific rather than shared. Finally, these differences widen as tasks grow harder: success rate is similar on easy tasks but separates sharply as exploration becomes more demanding. Our process-level analysis opens a new avenue in web agent evaluation, providing fine-grained and actionable insight into where and how each agent should be improved. Project page: https://jiwanchung.github.io/webstep
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。