让指令变动态推理路径,提升GUI操作的准确率
UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning
- 将指令视为多视角推理路径,动态选择最优执行思路
- 在多个基准上达到87.3%最高准确率,相对提升达76%
- 适合做GUI智能体、自动化测试与人机交互研究者
GUI grounding 是将自然语言指令映射到可操作的用户界面元素的核心能力。以往工作大多将指令视为静态意图代理,忽视了指令多样性和质量对定位性能的影响。通过对现有数据集的细致分析,我们发现其指令存在23.3%的错误率,并证明推理时利用指令多样性可带来高达76%的相对性能提升。本文提出 Instruction-as-Reasoning 范式,将指令视为动态分析路径,赋予模型在推理过程中选择最有效路径的能力。为此,我们设计两阶段训练框架:先在合成的多样化指令上进行监督微调(SFT),注入多视角推理能力;再通过强化学习(RL)优化路径选择与组合。所提模型 UI-Ins-7B 与 UI-Ins-32B 在五个挑战性基准上取得领先表现,展现出涌现式推理能力,能在推理时选择并合成新的指令路径。尤其,UI-Ins-32B 在 UI-I2E-Bench 上达到87.3%准确率,ScreenSpot-Pro 上为57.0%,MMBench-GUI L2 上为84.9%。此外,其作为执行器在 AndroidWorld 上实现74.1%的成功率。深入分析揭示了如何通过合理设计推理机制提升而非削弱定位效果,以及如何缓解 SFT+RL 框架中的策略坍塌问题。所有代码与模型权重将在 https://github.com/alibaba/UI-Ins 公开。
原文摘要 · Abstract (English)
GUI grounding, which maps natural-language instructions to actionable UI elements, is a core capability of GUI agents. Prior works largely treats instructions as a static proxy for user intent, overlooking the impact of instruction diversity and quality on grounding performance. Through a careful investigation of existing grounding datasets, we find a 23.3% flaw rate in their instructions and show that inference-time exploitation of instruction diversity yields up to a substantial 76% relative performance improvement. In this paper, we introduce the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives and enabling the model to select the most effective pathway during reasoning. To achieve this, we propose a two-stage training framework: supervised fine-tuning (SFT) on synthesized, diverse instructions to instill multi-perspective reasoning, followed by reinforcement learning (RL) to optimize pathway selection and composition. Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art results on five challenging grounding benchmarks and exhibit emergent reasoning, selectively composing and synthesizing novel instruction pathways at inference. In particular, UI-Ins-32B attains the best grounding accuracy, scoring 87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2. Furthermore, our model demonstrates strong agentic potential, achieving a 74.1% success rate on AndroidWorld using UI-Ins-7B as the executor. Our in-depth analysis reveals additional insights such as how reasoning can be formulated to enhance rather than hinder grounding performance, and how our method mitigates policy collapse in the SFT+RL framework. All code and model checkpoints will be publicly released in https://github.com/alibaba/UI-Ins.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。