让手机助手理解操作流程，出错能自动恢复。
Agent-SAMA: State-Aware Mobile Assistant
- 将界面当作状态，操作当作转移，构建动态状态机。
- 跨应用任务成功率84.0%，错误恢复率71.9%。
- 适合需要稳定执行的自动化手机操作场景。
移动图形用户界面（GUI）代理旨在根据用户指令在单个或多个应用内自主完成任务。尽管近期多模态大模型使这些代理能够解析界面并执行操作，但现有方法仍为被动响应式：仅基于当前界面进行推理，缺乏对应用导航流程的结构化表示，限制了其对执行上下文的理解、异常结果的检测及错误恢复能力。本文提出Agent-SAMA，一种状态感知的多代理框架，将应用执行建模为有限状态机（FSM），把界面视作状态，用户操作视为转移。Agent-SAMA部署四个专用代理，协同实时构建与使用FSM，指导任务规划、执行验证与恢复。在两类基准上评估：跨应用（Mobile-Eval-E、SPA-Bench）和单应用（AndroidWorld）。在Mobile-Eval-E上，任务成功率达84.0%，恢复率达71.9%；在SPA-Bench上，成功率为80.0%，恢复率为66.7%。相比基线，任务成功率提升最高达12%，恢复成功率提升13.8%。在AndroidWorld上，成功率达63.7%，优于基线。结果表明，结构化状态建模可显著提升鲁棒性，且可作为未来GUI代理的轻量级、模型无关记忆层。
原文摘要 · Abstract (English)
Mobile Graphical User Interface (GUI) agents aim to autonomously complete tasks within or across apps based on user instructions. While recent Multimodal Large Language Models (MLLMs) enable these agents to interpret UI screens and perform actions, existing agents remain fundamentally reactive. They reason over the current UI screen but lack a structured representation of the app navigation flow, limiting GUI agents' ability to understand execution context, detect unexpected execution results, and recover from errors. We introduce Agent-SAMA, a state-aware multi-agent framework that models app execution as a Finite State Machine (FSM), treating UI screens as states and user actions as transitions. Agent-SAMA implements four specialized agents that collaboratively construct and use FSMs in real time to guide task planning, execution verification, and recovery. We evaluate Agent-SAMA on two types of benchmarks: cross-app (Mobile-Eval-E, SPA-Bench) and mostly single-app (AndroidWorld). On Mobile-Eval-E, Agent-SAMA achieves an 84.0% success rate and a 71.9% recovery rate. On SPA-Bench, it reaches an 80.0% success rate with a 66.7% recovery rate. Compared to prior methods, Agent-SAMA improves task success by up to 12% and recovery success by 13.8%. On AndroidWorld, Agent-SAMA achieves a 63.7% success rate, outperforming the baselines. Our results demonstrate that structured state modeling enhances robustness and can serve as a lightweight, model-agnostic memory layer for future GUI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。