让AI自动探索新界面，提升跨环境操作能力。
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
- 用自主探索收集特定界面数据，持续优化模型
- 在新环境中准确率提升23.7%，超越基线方法
- 适合需要跨平台自动化任务的开发者使用
图形用户界面（GUI）操作定位是实现自动化的重要步骤，即将语言指令映射到界面可操作元素。现有方法依赖大规模GUI数据集微调多模态大模型（MLLM），但训练数据覆盖环境有限，导致模型在新环境性能下降。本文提出GUI-Bee，一种基于MLLM的自主代理，通过探索收集高质量、环境特异的数据，并持续微调模型以适应新环境。该代理采用新颖的Q值激励上下文强化学习（Q-ICRL）方法，提升探索效率与数据质量。同时，构建NovelScreenSpot基准测试，验证所收集数据的有效性。实验表明，GUI-Bee显著提升模型在新环境下的表现，且消融实验证实Q-ICRL能有效增强探索效率。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) action grounding is a critical step in GUI automation that maps language instructions to actionable elements on GUI screens. Most recent works of GUI action grounding leverage large GUI datasets to fine-tune MLLMs. However, the fine-tuning data always covers limited GUI environments, and we find the performance of the resulting model deteriorates in novel environments. We argue that the GUI grounding models should be further aligned to the novel environments to reveal their full potential, when the inference is known to involve novel environments, i.e., environments not used during the previous fine-tuning. To realize this, we first propose GUI-Bee, an MLLM-based autonomous agent, to collect high-quality, environment-specific data through exploration and then continuously fine-tune GUI grounding models with the collected data. Our agent leverages a novel Q-value-Incentive In-Context Reinforcement Learning (Q-ICRL) method to optimize exploration efficiency and data quality. Additionally, we introduce NovelScreenSpot, a benchmark for testing how well the data can help align GUI action grounding models to novel environments and demonstrate the effectiveness of data collected by GUI-Bee in the experiments. Furthermore, we conduct an ablation study to validate the Q-ICRL method in enhancing the efficiency of GUI-Bee. Project page: https://gui-bee.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。