用知识图谱增强GUI智能体决策，提升手机操作成功率。
KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation
- 将碎片化界面导航图转为向量库，支持实时检索。
- 任务成功率75.8%，比AutoDroid高8.9个百分点。
- 适合做移动应用自动化或智能助手的开发者参考。
尽管大型语言模型在图形用户界面（GUI）智能体中取得进展，但在复杂移动端任务中仍受限于缺乏特定应用知识。虽然界面转换图（UTGs）提供结构化导航表示，但因提取质量差和整合效率低而未被充分利用。本文提出KG-RAG框架，将零散的UTGs转化为结构化的向量数据库，实现高效实时检索。通过意图引导的LLM搜索方法，生成可执行的导航路径，显著提升智能体决策能力。在多种移动端应用上实验显示，KG-RAG达到75.8%的任务成功率（较AutoDroid提升8.9%）、84.6%的决策准确率（提升8.1%），平均任务步骤由4.5降至4.1。此外，构建了针对中文移动生态的KG-Android-Bench与KG-Harmony-Bench两个基准。该方法还可迁移至网页/桌面端（微博网页+40%成功率，QQ音乐桌面端+20%），且实验证明每款复杂应用约4小时的UTG构建成本即可达到性能饱和，支持实际部署权衡。
原文摘要 · Abstract (English)
Despite recent progress, Graphic User Interface (GUI) agents powered by Large Language Models (LLMs) struggle with complex mobile tasks due to limited app-specific knowledge. While UI Transition Graphs (UTGs) offer structured navigation representations, they are underutilized due to poor extraction and inefficient integration. We introduce KG-RAG, a Knowledge Graph-driven Retrieval-Augmented Generation framework that transforms fragmented UTGs into structured vector databases for efficient real-time retrieval. By leveraging an intent-guided LLM search method, KG-RAG generates actionable navigation paths, enhancing agent decision-making. Experiments across diverse mobile apps show that KG-RAG outperforms existing methods, achieving a 75.8% success rate (8.9% improvement over AutoDroid), 84.6% decision accuracy (8.1% improvement), and reducing average task steps from 4.5 to 4.1. Additionally, we present KG-Android-Bench and KG-Harmony-Bench, two benchmarks tailored to the Chinese mobile ecosystem for future research. Finally, KG-RAG transfers to web/desktop (+40% SR on Weibo-web; +20% on QQ Music-desktop), and a UTG cost ablation shows accuracy saturates at ~4h per complex app, enabling practical deployment trade-offs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。