arXiv:2507.16853cs.ROcs.MA2025-07被引 4

让手机助手自动执行复杂任务,出错能自纠,陌生环境会探索。

MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation

  • 分层自我反思机制,从单步操作到整体任务都能自检纠错。
  • 在真实安卓环境中达成62.9%的成功率,领先现有方法。
  • 适合需要稳定自动化操作的移动应用开发与测试场景。

多模态大模型的发展使得能够理解视觉输入并遵循用户指令的手机智能体成为可能,为自动化移动设备上的复杂任务开辟了新路径。然而,在真实移动场景中应用这些模型仍面临长周期任务执行、错误恢复困难以及陌生环境下的冷启动问题。为此,我们提出MobileUse,一个面向鲁棒且自适应移动任务执行的图形界面智能体。为提升长周期任务和动态环境中的韧性,我们引入分层自我反思架构,支持跨多时间尺度(从单个操作到整体任务)的自我监控、错误检测与恢复,同时通过按需反思策略保持效率。为解决冷启动问题,我们进一步设计主动探索模块,通过自主规划探索丰富对环境的理解。在AndroidWorld和AndroidLab基准上的评估表明,MobileUse达到62.9%和44.2%的成功率,刷新当前最佳性能。为推动实际应用,我们开源了可直接使用的物理手机自动化工具包,项目地址:https://github.com/MadeAgents/mobile-use。

原文摘要 · Abstract (English)

Recent advances in Multimodal Large Language Models (MLLMs) have enabled the development of mobile agents that can understand visual inputs and follow user instructions, unlocking new possibilities for automating complex tasks on mobile devices. However, applying these models to real-world mobile scenarios remains a significant challenge due to the long-horizon task execution, difficulty in error recovery, and the cold-start problem in unfamiliar environments. To address these challenges, we propose MobileUse, a GUI agent designed for robust and adaptive mobile task execution. To improve resilience in long-horizon tasks and dynamic environments, we introduce a hierarchical reflection architecture that enables the agent to self-monitor, detect, and recover from errors across multiple temporal scales-ranging from individual actions to overall task completion-while maintaining efficiency through a reflection-on-demand strategy. To tackle cold-start issues, we further introduce a proactive exploration module, which enriches the agent's understanding of the environment through self-planned exploration. Evaluations on AndroidWorld and AndroidLab benchmarks demonstrate that MobileUse establishes new state-of-the-art performance, achieving success rates of 62.9% and 44.2%, respectively. To facilitate real-world applications, we release an out-of-the-box toolkit for automated task execution on physical mobile devices, which is available at https://github.com/MadeAgents/mobile-use.

移动智能体自反思自动化测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。