arXiv:2503.21620cs.AI2025-03AAAI被引 152

用规则奖励强化学习,让AI更准预测手机操作。

UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning

论文配图:UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning
图 1 · 摘自论文原文
  • 设计规则奖励机制,用强化学习优化多模态模型的界面操作推理能力。
  • 在3个数据集上提升22.1%~12.7%准确率,接近更大模型性能。
  • 适合想提升GUI智能体控制效率的研究者与开发者。

近期的DeepSeek-R1通过基于规则的强化学习(RL)展示了语言模型中涌现的推理能力。尽管其在语言模型中表现成功,但在多模态领域,特别是图形用户界面(GUI)代理任务中的应用仍待探索。为此,我们提出UI-R1,首个探索基于规则的强化学习如何增强多模态大语言模型(MLLM)在GUI动作预测任务中推理能力的框架。具体而言,UI-R1引入一种新颖的基于规则的动作奖励机制,支持通过策略优化算法(如组相对策略优化,GRPO)进行模型优化。为实现高效训练,我们构建了一个包含136个高挑战性任务的小而高质量数据集,涵盖移动设备上的五种常见操作类型。实验结果表明,所提出的UI-R1-3B在域内(ID)和域外(OOD)任务上均显著优于基础模型(即Qwen2.5-VL-3B),在ScreenSpot、ScreenSpot-Pro和ANDROIDCONTROL上的平均准确率分别提升22.1%、6.0%和12.7%。此外,UI-R1-3B在76K样本监督微调的大模型(如OS-Atlas-7B)对比下表现具有竞争力。我们还开发了优化版本UI-R1-E-3B,显著提升了定位效率与准确率。这些结果证明了基于规则的强化学习在推动GUI理解与控制方面的潜力,为该领域未来研究铺平道路。代码与网站:https://github.com/lll6gg/UI-R1。

原文摘要 · Abstract (English)

The recent DeepSeek-R1 has showcased the emergence of reasoning capabilities in LLMs through reinforcement learning (RL) with rule-based rewards. Despite its success in language models, its application in multi-modal domains, particularly in graphic user interface (GUI) agent tasks, remains under-explored. To address this issue, we propose UI-R1, the first framework to explore how rule-based RL can enhance the reasoning capabilities of multimodal large language models (MLLMs) for GUI action prediction tasks. Specifically, UI-R1 introduces a novel rule-based action reward, enabling model optimization via policy-based algorithms such as Group Relative Policy Optimization (GRPO). For efficient training, we curate a small yet high-quality dataset of 136 challenging tasks, encompassing five common action types on mobile devices. Experimental results demonstrate that our proposed UI-R1-3B achieves significant improvements over the base model (i.e. Qwen2.5-VL-3B) on both in-domain (ID) and out-of-domain (OOD) tasks, with average accuracy gains of 22.1% on ScreenSpot, 6.0% on ScreenSpot-Pro, and 12.7% on ANDROIDCONTROL. Furthermore, UI-R1-3B delivers competitive performance compared to larger models (e.g., OS-Atlas-7B) trained via supervised fine-tuning (SFT) on 76K samples. We additionally develop an optimized version, UI-R1-E-3B, which significantly improves both grounding efficiency and accuracy. These results underscore the potential of rule-based reinforcement learning to advance GUI understanding and control, paving the way for future research in this domain. Code website: https://github.com/lll6gg/UI-R1.

GUI代理强化学习多模态动作预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。