arXiv:2505.16422cs.CV2025-05EMNLP被引 3

用世界模型提前规划,让手机自动操作更聪明。

Unlocking Smarter Device Control: Foresighted Planning with a World Model-Driven Code Execution Approach

  • 先构建可迭代优化的世界模型,理解全局环境
  • 通过代码执行实现44.4%相对成功率提升
  • 适合需要多步决策的自动化任务研究者

移动设备的自动化控制对于高效完成涉及多个顺序步骤的复杂任务至关重要。然而,由于每一步只能依赖视觉观察获取有限环境信息,当前方法通常采用仅关注即时观测的反应式策略,常导致次优决策。为此,我们提出「前瞻性规划与世界模型驱动代码执行(FPWC)」框架,通过在任务开始时构建面向任务、可迭代优化的‘世界模型’,强化智能体对环境的全局理解。随后,在该世界模型中进行迭代规划,生成前瞻性动作,并以可执行代码形式执行。在模拟环境和真实移动设备上的大量实验表明,该方法优于先前方法,尤其在模拟环境中相较最先进水平实现了44.4%的相对成功率提升。代码与演示视频见补充材料。

原文摘要 · Abstract (English)

The automatic control of mobile devices is essential for efficiently performing complex tasks that involve multiple sequential steps. However, these tasks pose significant challenges due to the limited environmental information available at each step, primarily through visual observations. As a result, current approaches, which typically rely on reactive policies, focus solely on immediate observations and often lead to suboptimal decision-making. To address this problem, we propose \textbf{Foresighted Planning with World Model-Driven Code Execution (FPWC)},a framework that prioritizes natural language understanding and structured reasoning to enhance the agent's global understanding of the environment by developing a task-oriented, refinable \emph{world model} at the outset of the task. Foresighted actions are subsequently generated through iterative planning within this world model, executed in the form of executable code. Extensive experiments conducted in simulated environments and on real mobile devices demonstrate that our method outperforms previous approaches, particularly achieving a 44.4\% relative improvement in task success rate compared to the state-of-the-art in the simulated environment. Code and demo are provided in the supplementary material.

智能控制世界模型代码执行移动自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。