为桌面操作智能体设计故障诊断修复框架，提升自动化任务成功率。
CUADebug: Diagnosing and Repairing Computer-Use Agent Failures
- 通过前后截图与操作日志对比，定位失败步骤的根因
- 在OSWorld基准上实现25.86%的任务持续成功，较基线翻倍
- 针对视觉感知、空间定位等多模态问题提供可执行修复建议
计算机使用智能体（CUAs）通过截图、鼠标键盘操作及状态化界面反馈控制真实桌面和网页界面，但其故障难以诊断与修复。不同于纯文本智能体，CUA故障源于视觉感知、空间定位、底层交互、任务推理与环境动态的耦合，构成独特的多模态因果定位难题。本文提出CUADebug框架，包含专用错误分类体系、人工标注的OSWorld失败基准（CUAErrorBench）以及工具增强型调试器CUADebugger。该调试器不一次性全轨迹提示，而是主动检查可疑步骤，结合前后截图与动作轨迹，输出包含根因步骤、错误类型、证据和修复策略的结构化诊断报告。对204条失败轨迹的人工标注显示，任务推理与控制类故障占比最高（110/204），其次为感知（36）、定位/交互（25）、外部/系统（13）及其他20例不可行任务。在主分组测试中，基于Gemini 2.5 Pro的CUADebugger将联合子类型与步骤诊断率从11.2%提升至19.6%，且在不同调试器底座下均表现一致。单次重执行评估中，基于根本原因分析（RCA）的条件使任务完成率达28.47%（机器生成）与29.90%（本方法），显著优于仅依赖历史的13.89%；在持续重执行场景中，本方法成功率由12.2%提升至25.86%，接近人类最优指导的29.21%。结果表明，根因诊断不仅能解释失败，还能提供可执行的修复信号。
原文摘要 · Abstract (English)
Computer-use agents (CUAs) operate real desktop and web interfaces through screenshots, mouse and keyboard actions, and stateful UI feedback, yet their failures remain difficult to diagnose and repair. Unlike text-only agents, CUA failures arise from coupled visual perception, spatial grounding, low-level interaction, task reasoning, and environment dynamics, making debugging a distinctive multimodal causal localization problem. We introduce CUADebug, a framework for diagnosing and repairing CUA failures. CUADebug includes a CUA-specific error taxonomy, CUAErrorBench, a human-annotated OSWorld failure benchmark, and CUADebugger, a tool-augmented debugger. Instead of prompting over the full trajectory once, CUADebugger actively inspects suspicious steps with paired before/after screenshots and action traces, then submits a structured diagnosis containing the root-cause step, error type, grounded evidence, and corrective strategy for re-execution. Human annotations over 204 failed trajectories show that task reasoning and control is the largest failure family (110/204), followed by perception (36), grounding/interaction (25), external/system (13), and an others category of 20 OSWorld infeasible-task cases. On the main Claude-agent split, CUADebugger improves joint subtype-and-step diagnosis from 11.2% to 19.6% with Gemini 2.5 Pro and improves consistently across debugger backbones. In single re-execution package evaluation, RCA-based conditions achieve higher task completion than history-only continuation (28.47% with machine RCA and 29.90% with our method, versus 13.89%); in continual re-execution, our method improves success from 12.2% to 25.86%, while human-oracle guidance reaches 29.21%. These results show that CUA root-cause diagnosis can provide actionable repair signals rather than merely post-hoc explanations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。