首个可操作图形界面的AI代理，实测展现端到端指令执行能力
The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use
- 通过设计多领域任务，验证模型从语言指令到桌面操作的端到端执行能力
- 在真实复杂环境中完成多种软件操作，展现出前所未有的自动化水平
- 提供开箱即用框架，助力开发者快速部署基于API的GUI自动化模型
最近发布的Claude 3.5 Computer Use是首个在公开测试版中提供图形界面操作能力的前沿AI模型。作为早期测试版本，其在真实复杂环境中的表现尚不明确。本案例研究通过精心设计的跨领域、多软件任务集，探索该模型的能力边界。观察结果表明，Claude 3.5 Computer Use具备前所未有的从自然语言指令到桌面操作的端到端执行能力。同时，本文提供了一个开箱即用的代理框架，支持基于API的GUI自动化模型快速部署。案例分析揭示了规划、动作与评估等关键问题，为未来改进提供方向。我们希望此初步探索能激发社区对GUI代理的深入研究。所有测试案例可通过项目仓库：https://github.com/showlab/computer_use_ootb 进行体验。
原文摘要 · Abstract (English)
The recently released model, Claude 3.5 Computer Use, stands out as the first frontier AI model to offer computer use in public beta as a graphical user interface (GUI) agent. As an early beta, its capability in the real-world complex environment remains unknown. In this case study to explore Claude 3.5 Computer Use, we curate and organize a collection of carefully designed tasks spanning a variety of domains and software. Observations from these cases demonstrate Claude 3.5 Computer Use's unprecedented ability in end-to-end language to desktop actions. Along with this study, we provide an out-of-the-box agent framework for deploying API-based GUI automation models with easy implementation. Our case studies aim to showcase a groundwork of capabilities and limitations of Claude 3.5 Computer Use with detailed analyses and bring to the fore questions about planning, action, and critic, which must be considered for future improvement. We hope this preliminary exploration will inspire future research into the GUI agent community. All the test cases in the paper can be tried through the project: https://github.com/showlab/computer_use_ootb.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。