用查询推理桥接界面定位与任务推理，小数据下性能媲美大模型。
Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks
- 通过截图和坐标推断用户查询，实现定位与推理对齐
- 仅用0.1%数据达到大模型水平，准确率显著提升
- 融合语义信息可进一步增强推理能力，适合资源受限场景
感知增强预训练，特别是通过定位技术，被广泛用于提升图形用户界面（GUI）代理的性能。然而，在资源受限场景中，坐标导向的定位与操作导向的推理之间存在格式差异，限制了定位对推理的有效性。为此，我们提出一种名为查询推理的查询导向枢纽方法，作为界面定位与推理之间的桥梁。通过从截图及其元素坐标中推断潜在用户查询，查询推理增强了对坐标的理解，并更贴近推理任务需求。实验表明，在相同训练数据规模下，查询推理优于以往定位技术。值得注意的是，查询推理在不足0.1%训练数据条件下，性能可媲美甚至超越大规模定位增强型OS-Atlas。此外，我们探讨了推理格式的影响，发现将额外语义信息融入输入可进一步提升推理表现。代码已公开于https://github.com/ZrW00/GUIPivot。
原文摘要 · Abstract (English)
Perception-enhanced pre-training, particularly through grounding techniques, is widely adopted to enhance the performance of graphical user interface (GUI) agents. However, in resource-constrained scenarios, the format discrepancy between coordinate-oriented grounding and action-oriented reasoning limits the effectiveness of grounding for reasoning tasks. To address this challenge, we propose a query-oriented pivot approach called query inference, which serves as a bridge between GUI grounding and reasoning. By inferring potential user queries from a screenshot and its associated element coordinates, query inference improves the understanding of coordinates while aligning more closely with reasoning tasks. Experimental results show that query inference outperforms previous grounding techniques under the same training data scale. Notably, query inference achieves comparable or even better performance to large-scale grounding-enhanced OS-Atlas with less than 0.1% of training data. Furthermore, we explore the impact of reasoning formats and demonstrate that integrating additional semantic information into the input further boosts reasoning performance. The code is publicly available at https://github.com/ZrW00/GUIPivot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。