arXiv:2410.00467cs.AIcs.HC2024-10EMNLP被引 24

让AI自动操作手机界面时能动态调整计划,提升准确率12.7%。

Dynamic Planning for LLM-based Graphical User Interface Automation

  • 根据环境反馈和操作历史动态更新计划,避免固定流程失效。
  • 在GPT-4V基础上准确率从34.66%提升至47.36%,提升12.7%。
  • 适合需要自适应决策的自动化任务,尤其对复杂界面有优势。

大型语言模型(LLMs)的兴起推动了基于LLM的自主智能体在智能手机图形用户界面(GUI)应用中的发展。当面对任务目标时,这些智能体通常模拟人类在GUI环境中执行操作直至完成。然而,如何设计有效计划以指导动作预测是关键挑战,尽管规划已被广泛认为能将复杂任务分解为一系列步骤。特别是,由于执行动作后环境GUI具有动态性,必须根据环境反馈和操作历史动态调整计划。我们发现,广泛使用的ReAct方法因历史对话过长而失效。为此,提出一种新方法D-PoT(Dynamic Planning of Thoughts),实现基于环境反馈和执行历史的动态规划。实验表明,D-PoT相比强基准GPT-4V准确率提升12.7%(34.66% → 47.36%)。分析显示,动态规划在不同主干LLM中具通用性,且有助于减少幻觉、适应未见任务。代码已开源:https://github.com/sqzhang-lazy/D-PoT。

原文摘要 · Abstract (English)

The advent of large language models (LLMs) has spurred considerable interest in advancing autonomous LLMs-based agents, particularly in intriguing applications within smartphone graphical user interfaces (GUIs). When presented with a task goal, these agents typically emulate human actions within a GUI environment until the task is completed. However, a key challenge lies in devising effective plans to guide action prediction in GUI tasks, though planning have been widely recognized as effective for decomposing complex tasks into a series of steps. Specifically, given the dynamic nature of environmental GUIs following action execution, it is crucial to dynamically adapt plans based on environmental feedback and action history.We show that the widely-used ReAct approach fails due to the excessively long historical dialogues. To address this challenge, we propose a novel approach called Dynamic Planning of Thoughts (D-PoT) for LLM-based GUI agents.D-PoT involves the dynamic adjustment of planning based on the environmental feedback and execution history. Experimental results reveal that the proposed D-PoT significantly surpassed the strong GPT-4V baseline by +12.7% (34.66% $\rightarrow$ 47.36%) in accuracy. The analysis highlights the generality of dynamic planning in different backbone LLMs, as well as the benefits in mitigating hallucinations and adapting to unseen tasks. Code is available at https://github.com/sqzhang-lazy/D-PoT.

GUI自动化动态规划LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。