用检索增强生成提升手机智能代理的准确性和记忆能力
MobileRAG: Enhancing Mobile Agent with Retrieval-Augmented Generation
- 引入RAG框架，通过外部知识检索辅助理解用户指令
- 在真实任务中比现有方法提升10.3%准确率，操作步骤更少
- 适合需要长期记忆和复杂交互的移动端AI助手研发者
智能手机已深度融入现代生活，基于大语言模型（LLMs）的移动代理逐渐普及。然而，现有代理存在三大问题：过度依赖LLM理解能力易出错；缺乏与外部环境交互，遇应用无法响应即终止；无记忆功能，每次需重构界面且无法纠正错误。为此，我们提出MobileRAG框架，融合InterRAG、LocalRAG与MemRAG，利用检索增强生成（RAG）技术更快更准地解析用户查询，完成复杂长序列操作。为全面评估性能，我们构建了更具挑战性的MobileRAG-Eval基准，包含大量需外部知识支持的真实世界任务。在该基准上的实验表明，MobileRAG能有效应对实际场景，相比顶尖方法提升10.3%准确率，且操作步骤更少。代码已开源。
原文摘要 · Abstract (English)
Smartphones have become indispensable in people's daily lives, permeating nearly every aspect of modern society. With the continuous advancement of large language models (LLMs), numerous LLM-based mobile agents have emerged. These agents are capable of accurately parsing diverse user queries and automatically assisting users in completing complex or repetitive operations. However, current agents 1) heavily rely on the comprehension ability of LLMs, which can lead to errors caused by misoperations or omitted steps during tasks, 2) lack interaction with the external environment, often terminating tasks when an app cannot fulfill user queries, and 3) lack memory capabilities, requiring each instruction to reconstruct the interface and being unable to learn from and correct previous mistakes. To alleviate the above issues, we propose MobileRAG, a mobile agents framework enhanced by Retrieval-Augmented Generation (RAG), which includes InterRAG, LocalRAG, and MemRAG. It leverages RAG to more quickly and accurately identify user queries and accomplish complex and long-sequence mobile tasks. Additionally, to more comprehensively assess the performance of MobileRAG, we introduce MobileRAG-Eval, a more challenging benchmark characterized by numerous complex, real-world mobile tasks that require external knowledge assistance. Extensive experimental results on MobileRAG-Eval demonstrate that MobileRAG can easily handle real-world mobile tasks, achieving 10.3\% improvement over state-of-the-art methods with fewer operational steps. Our code is publicly available at: https://github.com/liuxiaojieOutOfWorld/MobileRAG_arxiv
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。