arXiv:2505.14141cs.AI2025-05被引 4

用扩展有限状态机让手机智能体不迷路,自动规划操作路径

Building a Stable Planner: An Extended Finite State Machine Based Planning Module for Mobile GUI Agent

  • 用扩展有限状态机建模应用逻辑,拆解用户指令为可执行步骤
  • 在AndroidWorld上使任务成功率提升至63.8%,比无规划高28.8个百分点
  • 适合做手机自动化、AI助手的开发者,尤其关注落地可用性

移动端GUI智能体通过直接操作手机界面执行用户指令,具有显著提升用户体验的潜力。然而,这类智能体在任务规划上面临挑战,需持续分析界面并逐步生成操作指令,因缺乏对应用使用逻辑的深层理解,常导致执行过程中“迷失”。为此,我们提出SPlanner,一个即插即用的规划模块,用于生成指导视觉语言模型(VLM)执行任务的执行计划。该模块利用扩展有限状态机(EFSMs)建模移动应用的控制逻辑与配置,将用户指令分解为EFSM中定义的主要功能,并通过遍历EFSM生成执行路径。随后,借助大语言模型(LLM)将路径转化为自然语言计划,最终输出简洁可执行的指令,有效引导VLM生成交互式GUI操作以完成任务。SPlanner在反映真实移动使用场景的动态基准测试中表现优异。在AndroidWorld基准上,与Qwen2.5-VL-72B配合使用时,任务成功率达到63.8%,相比未使用规划辅助的版本提升28.8个百分点。

原文摘要 · Abstract (English)

Mobile GUI agents execute user commands by directly interacting with the graphical user interface (GUI) of mobile devices, demonstrating significant potential to enhance user convenience. However, these agents face considerable challenges in task planning, as they must continuously analyze the GUI and generate operation instructions step by step. This process often leads to difficulties in making accurate task plans, as GUI agents lack a deep understanding of how to effectively use the target applications, which can cause them to become "lost" during task execution. To address the task planning issue, we propose SPlanner, a plug-and-play planning module to generate execution plans that guide vision language model(VLMs) in executing tasks. The proposed planning module utilizes extended finite state machines (EFSMs) to model the control logits and configurations of mobile applications. It then decomposes a user instruction into a sequence of primary function modeled in EFSMs, and generate the execution path by traversing the EFSMs. We further refine the execution path into a natural language plan using an LLM. The final plan is concise and actionable, and effectively guides VLMs to generate interactive GUI actions to accomplish user tasks. SPlanner demonstrates strong performance on dynamic benchmarks reflecting real-world mobile usage. On the AndroidWorld benchmark, SPlanner achieves a 63.8% task success rate when paired with Qwen2.5-VL-72B as the VLM executor, yielding a 28.8 percentage point improvement compared to using Qwen2.5-VL-72B without planning assistance.

手机自动化状态机VLM任务规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。