arXiv:2603.07978cs.AI2026-03被引 7

让计算机代理通过探索学习专业级操作技能,效率接近人类专家。

OSExpert: Computer-Use Agents Learning Professional Skills via Exploration

  • 用图形界面深度优先搜索探索环境,自动生成任务技能库。
  • 在新测试中性能提升约20%,推理效率接近人类80%。
  • 适合需要精细操作和跨界面适应的自动化系统研发者。

通用型计算机使用代理在多种数字环境中表现出色,但我们的新基准测试OSExpert-Eval显示,它们仍远不如人类专家实用。尽管推理时扩展能实现适应,这些代理在完成复杂任务时效率低下、性能下降,难以迁移到未见过的用户界面,且对细粒度操作序列处理困难。为此,我们提出基于GUI的深度优先搜索(GUI-DFS)探索算法,全面探测并验证环境中的基本功能单元。代理随后利用单元技能间的组合性,自建复合任务课程。为支持细粒度动作,我们构建了动作原语数据库,代理在探索中发现后将其保存为技能集。利用所学技能,代理性能与效率得以提升:(1) 赋予代理现成的程序知识,使长轨迹仅需一次规划即可准确生成动作;(2) 提前终止推理时扩展,明确自身能力边界。大量实验表明,该环境学习代理在迈向专家级计算机操作上迈出重要一步,在OSExpert-Eval上性能提升约20%,效率差距缩小约80%。

原文摘要 · Abstract (English)

General-purpose computer-use agents have shown impressive performance across diverse digital environments. However, our new benchmark, OSExpert-Eval, indicates they remain far less helpful than human experts. Although inference-time scaling enables adaptation, these agents complete complex tasks inefficiently with degraded performance, transfer poorly to unseen UIs, and struggle with fine-grained action sequences. To solve the problem, we introduce a GUI-based depth-first search (GUI-DFS) exploration algorithm to comprehensively explore and verify an environment's unit functions. The agent then exploits compositionality between unit skills to self-construct a curriculum for composite tasks. To support fine-grained actions, we curate a database of action primitives for agents to discover during exploration; these are saved as a skill set once the exploration is complete. We use the learned skills to improve the agent's performance and efficiency by (1) enriching agents with ready-to-use procedural knowledge, allowing them to plan only once for long trajectories and generate accurate actions, and (2) enabling them to end inference-time scaling earlier by realizing their boundary of capabilities. Extensive experiments show that our environment-learned agent takes a meaningful step toward expert-level computer use, achieving a around 20 percent performance gain on OSExpert-Eval and closing the efficiency gap to humans by around 80 percent

计算机代理技能学习效率提升用户界面

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。