arXiv:2504.10127cs.AIcs.CL2025-04被引 11

用丰富推理数据训练模型,突破GUI自动化中的数据瓶颈。

Breaking the Data Barrier -- Building GUI Agents Through Task Generalization

  • 在中期训练阶段引入多模态与文本推理任务,提升模型泛化能力。
  • 纯文本数学数据使网页代理性能提升5.6%,安卓代理提升5.4%。
  • 发现感知数据作用有限,提出最优混合数据方案,显著提升效果。

图形用户界面(GUI)代理为跨平台数字任务自动化提供了潜力巨大的解决方案,但其性能常受限于高质量轨迹数据的稀缺。为解决此问题,我们提出在特定中期训练阶段,利用数据丰富、推理密集的任务训练视觉语言模型(VLM),并探究这些任务如何促进对GUI规划场景的泛化。具体而言,我们探索了多种具备充足指令微调数据的任务,包括GUI感知、多模态推理和文本推理。在11项中等训练任务上的广泛实验表明:(1) 任务泛化效果显著,在多数设置中带来明显提升。例如,多模态数学推理使AndroidWorld上性能提升6.3%;值得注意的是,仅使用文本数学数据即显著提升网页代理表现,分别在WebArena和AndroidWorld上取得5.6%和5.4%的绝对提升,证明了从文本到视觉域的显著跨模态泛化;(2) 与以往假设相反,此前被认为与GUI任务高度相关且广泛使用的GUI感知数据,对最终性能影响相对有限;(3) 基于上述发现,我们识别出最有效的中等训练任务,并构建优化混合数据集,在WebArena上实现8.0%的绝对提升,AndroidWorld上达12.2%。本工作为GUI代理的跨域知识迁移提供了重要洞见,并提出应对该领域数据稀缺问题的实用方法。代码、数据与模型将开源于https://github.com/hkust-nlp/GUIMid。

原文摘要 · Abstract (English)

Graphical User Interface (GUI) agents offer cross-platform solutions for automating complex digital tasks, with significant potential to transform productivity workflows. However, their performance is often constrained by the scarcity of high-quality trajectory data. To address this limitation, we propose training Vision Language Models (VLMs) on data-rich, reasoning-intensive tasks during a dedicated mid-training stage, and then examine how incorporating these tasks facilitates generalization to GUI planning scenarios. Specifically, we explore a range of tasks with readily available instruction-tuning data, including GUI perception, multimodal reasoning, and textual reasoning. Through extensive experiments across 11 mid-training tasks, we demonstrate that: (1) Task generalization proves highly effective, yielding substantial improvements across most settings. For instance, multimodal mathematical reasoning enhances performance on AndroidWorld by an absolute 6.3%. Remarkably, text-only mathematical data significantly boosts GUI web agent performance, achieving a 5.6% improvement on WebArena and 5.4% improvement on AndroidWorld, underscoring notable cross-modal generalization from text-based to visual domains; (2) Contrary to prior assumptions, GUI perception data - previously considered closely aligned with GUI agent tasks and widely utilized for training - has a comparatively limited impact on final performance; (3) Building on these insights, we identify the most effective mid-training tasks and curate optimized mixture datasets, resulting in absolute performance gains of 8.0% on WebArena and 12.2% on AndroidWorld. Our work provides valuable insights into cross-domain knowledge transfer for GUI agents and offers a practical approach to addressing data scarcity challenges in this emerging field. The code, data and models will be available at https://github.com/hkust-nlp/GUIMid.

GUI代理多模态泛化能力数据稀缺

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。