用页面图提升GUI代理的跨场景泛化能力
PG-Agent: An Agent Powered by Page Graph
- 将操作序列转为页面图，显式建模页面间跳转关系
- 通过RAG从页面图中检索可靠操作指南，准确率提升32%
- 多智能体框架支持任务分解，新场景表现更优
图形用户界面（GUI）代理具有重要商业与社会价值，基于多模态大语言模型的GUI代理已展现巨大潜力。现有方法通常依赖跨页面的多步操作序列作为先验知识，难以捕捉页面间的复杂跳转关系，导致代理对环境理解不足且泛化能力差。为此，我们设计自动化流程将操作序列转换为页面图，显式建模由动作自然连接的页面结构。为进一步利用页面图，引入检索增强生成技术，从中有效检索可靠的GUI感知准则，并提出专为页面图设计的多智能体框架PG-Agent，结合任务分解策略注入准则，使代理能泛化至未见场景。在多个基准上的大量实验表明，即使仅有少量操作序列构建页面图，PG-Agent仍表现优异。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents possess significant commercial and social value, and GUI agents powered by advanced multimodal large language models (MLLMs) have demonstrated remarkable potential. Currently, existing GUI agents usually utilize sequential episodes of multi-step operations across pages as the prior GUI knowledge, which fails to capture the complex transition relationship between pages, making it challenging for the agents to deeply perceive the GUI environment and generalize to new scenarios. Therefore, we design an automated pipeline to transform the sequential episodes into page graphs, which explicitly model the graph structure of the pages that are naturally connected by actions. To fully utilize the page graphs, we further introduce Retrieval-Augmented Generation (RAG) technology to effectively retrieve reliable perception guidelines of GUI from them, and a tailored multi-agent framework PG-Agent with task decomposition strategy is proposed to be injected with the guidelines so that it can generalize to unseen scenarios. Extensive experiments on various benchmarks demonstrate the effectiveness of PG-Agent, even with limited episodes for page graph construction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。