用双层知识检索提升手机多智能体长任务成功率
Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automation
- 分层设计:规划用人类验证的任务计划,执行用精准的界面操作指导
- 实测任务完成率提升11.0%,步骤效率提高10.2%
- 适合做跨应用、长周期手机自动化研究与开发
移动智能体在现实世界中执行长周期、跨应用任务时,现有最先进(SoTA)方法的成功率仍不理想。我们归因于其过度依赖大语言模型内部的静态知识,导致两个关键失败点:高层规划中的策略幻觉和低层界面操作中的执行错误。本文核心洞察是:高层规划与底层操作需要不同类型的认知知识——前者需高阶策略经验,后者需紧扣具体应用界面的精确指令。为此,我们提出Mobile-Agent-RAG,一种创新的分层多智能体框架,融合双层检索增强机制。规划阶段引入Manager-RAG,通过检索人类验证的完整任务计划,减少策略幻觉;执行阶段设计Operator-RAG,检索与当前应用和子任务匹配的精确原子动作指导,提升操作准确率。为精准交付两类知识,构建了两个专用检索知识库。此外,提出Mobile-Eval-RAG,一个用于评估此类智能体在真实多应用、长周期任务上的挑战性基准。大量实验表明,Mobile-Agent-RAG显著优于现有基线,任务完成率提升11.0%,步骤效率提高10.2%,建立了一种上下文感知、可靠的多智能体移动自动化新范式。
原文摘要 · Abstract (English)
Mobile agents show immense potential, yet current state-of-the-art (SoTA) agents exhibit inadequate success rates on real-world, long-horizon, cross-application tasks. We attribute this bottleneck to the agents' excessive reliance on static, internal knowledge within MLLMs, which leads to two critical failure points: 1) strategic hallucinations in high-level planning and 2) operational errors during low-level execution on user interfaces (UI). The core insight of this paper is that high-level planning and low-level UI operations require fundamentally distinct types of knowledge. Planning demands high-level, strategy-oriented experiences, whereas operations necessitate low-level, precise instructions closely tied to specific app UIs. Motivated by these insights, we propose Mobile-Agent-RAG, a novel hierarchical multi-agent framework that innovatively integrates dual-level retrieval augmentation. At the planning stage, we introduce Manager-RAG to reduce strategic hallucinations by retrieving human-validated comprehensive task plans that provide high-level guidance. At the execution stage, we develop Operator-RAG to improve execution accuracy by retrieving the most precise low-level guidance for accurate atomic actions, aligned with the current app and subtask. To accurately deliver these knowledge types, we construct two specialized retrieval-oriented knowledge bases. Furthermore, we introduce Mobile-Eval-RAG, a challenging benchmark for evaluating such agents on realistic multi-app, long-horizon tasks. Extensive experiments demonstrate that Mobile-Agent-RAG significantly outperforms SoTA baselines, improving task completion rate by 11.0% and step efficiency by 10.2%, establishing a robust paradigm for context-aware, reliable multi-agent mobile automation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。