arXiv:2601.04035cs.AI2026-01被引 14

用文字草图建模预测操作后果,提升手机自动化代理的长程决策能力。

MobileDreamer: Generative Sketch World Model for GUI Agent

  • 将界面图像转为关键元素草图,保留空间信息进行状态预测
  • 在Android World上任务成功率提升5.25%,达当前最优
  • 适合需长远规划的移动端自动化场景

移动GUI代理在现实自动化中展现出巨大潜力,但多数现有代理仍为反应式,仅依赖当前屏幕,限制了其在长时序任务中的表现。通过重复交互构建世界模型,可预测操作结果并支持更优决策,但挑战在于模型需具备空间感知能力的同时保持高效性。本文提出MobileDreamer,一种基于世界模型的前瞻框架,配备文本草图世界模型与滚动想象策略。文本草图世界模型通过学习将数字图像转化为关键任务相关草图,并设计新型无序不变学习策略以保留GUI元素的空间信息;滚动想象策略则利用世界模型的预测能力优化动作选择过程。在Android World上的实验表明,MobileDreamer达到当前最优性能,任务成功率提升5.25%。世界模型评估进一步验证了文本草图建模对关键GUI元素的准确预测能力。

原文摘要 · Abstract (English)

Mobile GUI agents have shown strong potential in real-world automation and practical applications. However, most existing agents remain reactive, making decisions mainly from current screen, which limits their performance on long-horizon tasks. Building a world model from repeated interactions enables forecasting action outcomes and supports better decision making for mobile GUI agents. This is challenging because the model must predict post-action states with spatial awareness while remaining efficient enough for practical deployment. In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. It consists of textual sketch world model and rollout imagination for GUI agent. Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. The rollout imagination strategy for GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%. World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements.

GUI代理世界模型草图生成移动自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。