D-Artemis让手机界面智能体像人一样思考、校准、反思,显著提升任务成功率。
D-Artemis: A Deliberative Cognitive Framework for Mobile GUI Multi-Agents
- 借鉴人类认知循环,分三步决策:思考、对齐、反思
- 在安卓世界和屏点数据集上分别达75.8%和96.8%成功率
- 无需复杂训练数据,适合通用手机自动化场景
图形用户界面(GUI)智能体旨在通过模拟用户操作自动化各类任务。尽管进展迅速,现有方法仍面临端到端训练的数据瓶颈、错误检测延迟高及指导冲突等挑战。受人类认知循环——思考、对齐、反思的启发,本文提出D-Artemis:一种新型思辨式框架。该框架采用细粒度、应用特定的提示检索机制辅助决策,并引入主动预执行对齐阶段,通过思维-动作一致性(TAC)检查模块与动作修正代理(ACA)协同降低执行失败风险。后执行状态反思代理(SRA)完成认知闭环,实现经验驱动的战略学习。关键在于,D-Artemis无需在复杂轨迹数据集上训练,即可增强通用多模态大语言模型(MLLM)在GUI任务中的能力,展现出强泛化性。在两大主流基准上均达到新SOTA,AndroidWorld成功率达75.8%,ScreenSpot-V2达96.8%。大量消融实验进一步验证了各组件的关键贡献。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents aim to automate a wide spectrum of human tasks by emulating user interaction. Despite rapid advancements, current approaches are hindered by several critical challenges: data bottleneck in end-to-end training, high cost of delayed error detection, and risk of contradictory guidance. Inspired by the human cognitive loop of Thinking, Alignment, and Reflection, we present D-Artemis -- a novel deliberative framework in this paper. D-Artemis leverages a fine-grained, app-specific tip retrieval mechanism to inform its decision-making process. It also employs a proactive Pre-execution Alignment stage, where Thought-Action Consistency (TAC) Check module and Action Correction Agent (ACA) work in concert to mitigate the risk of execution failures. A post-execution Status Reflection Agent (SRA) completes the cognitive loop, enabling strategic learning from experience. Crucially, D-Artemis enhances the capabilities of general-purpose Multimodal large language models (MLLMs) for GUI tasks without the need for training on complex trajectory datasets, demonstrating strong generalization. D-Artemis establishes new state-of-the-art (SOTA) results across both major benchmarks, achieving a 75.8% success rate on AndroidWorld and 96.8% on ScreenSpot-V2. Extensive ablation studies further demonstrate the significant contribution of each component to the framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。