让大模型精准点击界面元素，提升自动化工具效率
ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
- 大模型负责思考规划，专用模型专注定位界面元素
- 在AITW基准上任务成功率超越CogAgent和AppAgent
- 支持模拟器与真实手机测试，适合开发智能助手的人
随着计算机和智能手机等图形化设备的普及，高效自动化工具的需求日益迫切。尽管多模态大语言模型（如GPT-4V）在诸多领域表现优异，但在图形用户界面（GUI）交互方面仍存在困难，限制了其在日常任务自动化中的应用。本文提出ClickAgent框架，将推理与行动规划交由多模态大模型处理，同时引入独立的UI定位模型（如SeeClick）精确识别屏幕上的目标界面元素。该设计有效解决当前大模型在界面元素定位上的关键缺陷。我们在Android手机模拟器和真实设备上进行了评估，采用任务成功率作为核心指标。结果表明，ClickAgent在AITW基准上优于其他基于提示的自主代理（如CogAgent、AppAgent）。
原文摘要 · Abstract (English)
With the growing reliance on digital devices equipped with graphical user interfaces (GUIs), such as computers and smartphones, the need for effective automation tools has become increasingly important. While multimodal large language models (MLLMs) like GPT-4V excel in many areas, they struggle with GUI interactions, limiting their effectiveness in automating everyday tasks. In this paper, we introduce ClickAgent, a novel framework for building autonomous agents. In ClickAgent, the MLLM handles reasoning and action planning, while a separate UI location model (e.g., SeeClick) identifies the relevant UI elements on the screen. This approach addresses a key limitation of current-generation MLLMs: their difficulty in accurately locating UI elements. ClickAgent outperforms other prompt-based autonomous agents (CogAgent, AppAgent) on the AITW benchmark. Our evaluation was conducted on both an Android smartphone emulator and an actual Android smartphone, using the task success rate as the key metric for measuring agent performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。