让GUI智能体能主动回忆过去界面,解决长期任务记忆难题
PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents
- 用双层摘要+主动检索机制,按需调用历史截图
- 在8.6K样本上训练,移动端导航成功率显著提升
- 无需额外训练就能跨领域应用,适合长流程自动化
基于多模态大模型的图形用户界面(GUI)代理有望实现类人交互,但长周期任务仍受制于记忆能力。现有方法或截断历史,或依赖简单文本摘要,易丢失关键视觉信息。本文提出PAL-UI(规划中的主动回溯),通过双层摘要代理捕捉观测与动作层面的线索,并配备专用检索工具,在规划时主动召回特定历史截图。我们在移动界面导航轨迹中构建了包含8.6K样本的逐步指令数据集,基于Qwen2.5-VL训练出PAL-UI-3B和PAL-UI-7B模型。大量实验表明,该框架在移动端导航任务中显著优于基线模型,即使在数据高效设置下也表现优异;且具备强跨域泛化能力,在网页导航中无需额外训练即取得显著提升。本工作凸显了主动记忆检索对视觉驱动GUI代理长程规划能力的关键作用。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) promise human-like interaction with software applications, yet long-horizon tasks remain challenging due to memory limitations. Existing approaches either truncate history or rely on simple textual summaries, which risk losing critical information when past visual details become necessary for future decisions. In this paper, we propose \textbf{PAL-UI} (\textbf{P}lanning with \textbf{A}ctive \textbf{L}ook-back), a novel framework that enables GUI agents to adaptively retrieve past observations when required. PAL-UI combines a dual-level summarization agent, capturing both observation-level cues and action-level outcomes, with a dedicated retrieval tool that allows the agent to recall specific historical screenshots during planning. We curate a step-level instruction dataset of 8.6K samples from mobile GUI navigation trajectories and train \textbf{PAL-UI-3B} and \textbf{PAL-UI-7B} models based on Qwen2.5-VL. Extensive experiments demonstrate that PAL-UI significantly outperforms baseline models and prior methods in mobile GUI navigation tasks, even under data-efficient settings. Moreover, PAL-UI exhibits strong cross-domain generalization, achieving notable improvements in web navigation without additional training. Our work highlights the potential of active memory retrieval for long-horizon planning capabilities of vision-based GUI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。