arXiv:2510.17790cs.CVcs.CL2025-10被引 25

让电脑操作智能体同时用鼠标键盘和工具接口,更稳更快地完成任务。

UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action

  • 融合鼠标点击等基础操作与高阶工具调用,实现混合动作决策。
  • 在OSWorld上比现有方法快11%、成功率提升22%,跨域验证成功率达21.7%。
  • 适合需要稳定执行复杂任务的自动化系统开发者使用。

计算机使用智能体受限于仅能执行原始GUI操作(如点击、输入、滚动),导致执行链脆弱易崩溃。尽管基于API的智能体可通过结构化接口调用丰富能力,但计算机使用智能体仍局限于低层视觉交互。本文提出UltraCUA——一个通过混合动作无缝融合原始GUI操作与高层工具执行的基础模型。创新包含四方面:第一,自动化流水线从软件文档和代码库中提取并扩展工具能力;第二,合成数据引擎生成超过17,000个可验证的任务,涵盖真实世界复杂性;第三,全面收集混合动作轨迹,整合GUI原语与策略性工具调用;第四,采用两阶段训练法,结合监督微调与在线强化学习,实现对GUI与API动作的智能选择。在7B与32B版本的评估中,UltraCUA在OSWorld上相对现有方法提升22%成功率,平均执行速度加快11%;在WindowsAgentArena上的跨域验证显示21.7%的成功率,优于基于Windows训练的基线模型。混合动作范式有效减少错误传播,提升执行效率。该工作建立了一种可扩展的范式,连接原始视觉交互与高层工具智能,使计算机使用智能体在多样环境与复杂任务中更具鲁棒性与适应性。

原文摘要 · Abstract (English)

Computer-use agents face a fundamental limitation. They rely exclusively on primitive GUI actions (click, type, scroll), creating brittle execution chains prone to cascading failures. While API-driven agents harness rich capabilities through structured interfaces and tools, computer-use agents remain constrained to low-level visual interactions. We present UltraCUA, a foundation model that transcends this limitation through hybrid action-seamlessly unifying primitive GUI operations with high-level tool execution. Our innovation rests on four critical advances. First, an automated pipeline extracts and scales tool capabilities from software documentation and code repositories. Second, a synthetic data engine produces 17,000+ verifiable tasks capturing real-world computer-use complexity. Third, comprehensive hybrid action trajectory collection incorporates both GUI primitives and strategic tool calls. Fourth, a two-stage training methodology combines supervised fine-tuning with online reinforcement learning, enabling intelligent action selection between GUI and API. Evaluation with our 7B and 32B UltraCUA models reveals transformative performance gains. On OSWorld, UltraCUA achieves 22% relative improvement while executing 11% faster than existing approaches, averagely. Cross-domain validation on WindowsAgentArena demonstrates robust generalization with 21.7% success rate, surpassing Windows-trained baselines. The hybrid action paradigm proves essential, reducing error propagation while improving execution efficiency. This work establishes a scalable paradigm bridging primitive GUI interactions and high-level tool intelligence, enabling more resilient and adaptable computer use agents for diverse environments and complex real-world tasks.

智能体混合动作自动化基础模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。