让手机端AI助手更快决策,通过提前探索界面元素减少推理步骤。
MobileExplorer: Accelerating On-Device Inference for Mobile GUI Agents via Online Exploration

- 用轻量并行探索界面元素,利用视觉语言模型的长推理时间。
- 平均减少23%推理步数和端到端延迟,任务成功率最高提升5%。
- 适合注重隐私与低延迟的移动端AI应用开发者。
移动图形用户界面(GUI)代理使AI模型能自主操作智能手机。但现有系统多聚焦任务准确率,依赖云端模型推理,带来隐私风险和网络延迟。因此,完全在设备端部署仍不充分。我们提出MobileExplorer,通过在线探索加速基于视觉的移动GUI代理的设备端推理。核心思路是利用视觉语言模型(VLMs)每步推理时间较长的特点,对语义相关的界面元素进行轻量、并行探索。推理过程中,代理主动探测相关界面元素,并将探索轨迹记录为结构化记忆。为确保真实移动环境下的可靠执行,设计两级回滚机制,在快速但粗略的回溯策略失败时恢复初始界面状态。收集的探索轨迹被总结为简洁上下文提示,注入后续推理中以增强效果。我们在多个现成设备上使用AndroidWorld基准及新设计的复杂任务和动态设备环境评估MobileExplorer。结果表明,其平均减少23%推理步数和端到端延迟,同时任务成功率维持或提升最多5%。真实世界表现视频演示见https://youtu.be/thK7MJmdlvM。
原文摘要 · Abstract (English)
Mobile graphical user interface (GUI) agents enable AI models to autonomously operate smartphones on behalf of users. However, most existing systems focus primarily on optimizing task accuracy and rely on cloud-hosted models for inference, which introduces privacy concerns and network-dependent latency. As a result, fully on-device deployment of mobile GUI agents remains underexplored. We propose MobileExplorer, a new framework that accelerates on-device inference for vision-based mobile GUI agents via online exploration. The key idea is to exploit the long per-step reasoning time of vision-language models (VLMs) by performing lightweight, parallel exploration of UI elements. During model inference, the agent proactively probes semantically relevant UI elements and records these exploration traces as structured memory. To ensure reliable execution in live mobile environments, we design a two-level rollback mechanism that robustly restores the initial UI state when a fast but naive backtracking strategy fails. The collected exploration traces are then summarized into concise contextual hints and injected into the prompt to enhance the subsequent reasoning step. We evaluate MobileExplorer on multiple off-the-shelf devices using the AndroidWorld benchmark, as well as newly designed, more complex tasks and dynamic on-device environments. MobileExplorer reduces the average number of reasoning steps and end-to-end latency by 23\%, while maintaining or improving task success rates by up to 5\%. A video demonstration of MobileExplorer performance in the real world is available at https://youtu.be/thK7MJmdlvM .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。