Agent S让AI像人一样操作电脑，自动完成复杂多步骤任务。
Agent S: An Open Agentic Framework that Uses Computers Like a Human
- 通过多层级经验增强的分层规划，结合内外知识学习
- 在OSWorld上成功率达83.6%相对提升，创新高
- 适配不同系统，适合自动化测试与智能助手研究者
我们提出Agent S，一个开源的自主代理框架，可通过图形用户界面（GUI）实现对计算机的自主交互，旨在通过自动化复杂、多步骤任务来革新人机交互方式。Agent S针对自动化计算机任务中的三大挑战：获取领域特定知识、长程任务规划以及处理动态非统一界面，引入了经验增强的分层规划机制，在多个层级上融合外部知识搜索与内部经验检索，提升任务规划与子任务执行效率。此外，它采用代理-计算机界面（ACI），基于多模态大语言模型（MLLMs）更好地激发GUI代理的推理与控制能力。在OSWorld基准上的评估显示，Agent S相比基线成功率提升9.37%（相对提升83.6%），达到新最佳性能。综合分析验证了各组件有效性，并为未来改进提供洞见。此外，Agent S在新发布的WindowsAgentArena基准上展现出广泛泛化能力。代码已开源：https://github.com/simular-ai/Agent-S。
原文摘要 · Abstract (English)
We present Agent S, an open agentic framework that enables autonomous interaction with computers through a Graphical User Interface (GUI), aimed at transforming human-computer interaction by automating complex, multi-step tasks. Agent S aims to address three key challenges in automating computer tasks: acquiring domain-specific knowledge, planning over long task horizons, and handling dynamic, non-uniform interfaces. To this end, Agent S introduces experience-augmented hierarchical planning, which learns from external knowledge search and internal experience retrieval at multiple levels, facilitating efficient task planning and subtask execution. In addition, it employs an Agent-Computer Interface (ACI) to better elicit the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves a new state-of-the-art. Comprehensive analysis highlights the effectiveness of individual components and provides insights for future improvements. Furthermore, Agent S demonstrates broad generalizability to different operating systems on a newly-released WindowsAgentArena benchmark. Code available at https://github.com/simular-ai/Agent-S.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。