构建可复用的计算机操作技能库,提升智能体完成复杂任务的能力
CUA-Skill: Develop Skills for Computer Using Agent
- 将人类操作电脑的技能结构化为可组合的参数化模块
- 在WindowsAgentArena上实现57.5%的成功率,优于现有方法
- 适合研究智能体自动化与人机交互的开发者和研究人员
计算机使用智能体(CUAs)旨在自主操作计算机系统以完成现实任务。然而,现有智能体系统难以扩展,且性能仍远低于人类。核心瓶颈在于缺乏可复用、结构化的技能抽象,无法有效捕捉人类与图形界面的交互方式及其利用路径。本文提出CUA-Skill,一个大规模的计算机使用技能库,将人类操作知识编码为带有参数化执行与组合图的技能。该技能库覆盖常见Windows应用,构成可扩展、可靠的智能体开发基础设施。基于此,我们构建了CUA-Skill Agent,支持动态技能检索、参数实例化与记忆感知的容错恢复。实验表明,CUA-Skill显著提升了端到端智能体在挑战性基准上的成功率与鲁棒性。在WindowsAgentArena上,其最佳三次运行成功率达57.5%,且效率显著优于先前及同期方法。
原文摘要 · Abstract (English)
Computer-Using Agents (CUAs) aim to autonomously operate computer systems to complete real-world tasks. However, existing agentic systems remain difficult to scale and lag behind human performance. A key limitation is the absence of reusable and structured skill abstractions that capture how humans interact with graphical user interfaces and how to leverage these skills. We introduce CUA-Skill, a computer-using agentic skill base that encodes human computer-use knowledge as skills coupled with parameterized execution and composition graphs. CUA-Skill is a large-scale library of carefully engineered skills spanning common Windows applications, serving as a practical infrastructure and tool substrate for scalable, reliable agent development. Built upon this skill base, we construct CUA-Skill Agent, an end-to-end computer-using agent that supports dynamic skill retrieval, argument instantiation, and memory-aware failure recovery. Our results demonstrate that CUA-Skill substantially improves execution success rates and robustness on challenging end-to-end agent benchmarks, establishing a strong foundation for future computer-using agent development. On WindowsAgentArena, CUA-Skill Agent achieves state-of-the-art 57.5% (best of three) successful rate while being significantly more efficient than prior and concurrent approaches. The project page is available at https://microsoft.github.io/cua_skill/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。