让AI通过写代码和操作界面协同完成复杂任务,效率提升明显。
CoAct-1: Computer-using Multi-Agent System with Coding Actions
- 用编程与界面操作混合执行任务,动态分配给不同智能体。
- 在OSWorld上成功率达60.76%,平均步骤数降至10.15,优于旧方法。
- 适合需要高效自动化处理文件、数据的开发者或研究者。
仅依赖图形用户界面(GUI)操作的自主智能体在处理复杂、长周期任务时效率低且易出错。本文提出CoAct-1,一种结合GUI控制与程序执行的多智能体系统。其核心是调度器,可将子任务动态分配给传统GUI操作员或专用程序员智能体,后者能编写并运行Python或Bash脚本。该混合方式使智能体跳过繁琐的GUI操作,如文件管理与数据处理,仅在必要时使用视觉交互。在挑战性OSWorld基准测试中,CoAct-1达到60.76%的新高成功率,平均任务步数仅为10.15,显著低于领先GUI方法的15步。结果表明,将编程作为核心动作可实现更强大、高效且可扩展的通用计算机自动化。
原文摘要 · Abstract (English)
Autonomous agents that operate computers via Graphical User Interfaces (GUIs) often struggle with efficiency and reliability on complex, long-horizon tasks. While augmenting these agents with planners can improve task decomposition, they remain constrained by the inherent limitations of performing all actions through GUI manipulation, leading to brittleness and inefficiency. In this work, we introduce a more robust and flexible paradigm: enabling agents to use coding as a enhanced action. We present CoAct-1, a novel multi-agent system that synergistically combines GUI-based control with direct programmatic execution. CoAct-1 features an Orchestrator that dynamically delegates subtasks to either a conventional GUI Operator or a specialized Programmer agent, which can write and execute Python or Bash scripts. This hybrid approach allows the agent to bypass inefficient GUI action sequences for tasks like file management and data processing, while still leveraging visual interaction when necessary. We evaluate our system on the challenging OSWorld benchmark, where CoAct-1 achieves a new state-of-the-art success rate of 60.76%, significantly outperforming prior methods. Furthermore, our approach dramatically improves efficiency, reducing the average number of steps required to complete a task to just 10.15, compared to 15 for leading GUI agents. Our results demonstrate that integrating coding as a core action provides a more powerful, efficient, and scalable path toward generalized computer automation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。