用单次交互生成精准点击指导,提升GUI自动操作的效率与准确率
Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding

- 基于单次交互构建教师模型,提供密集的点击位置指导
- 在六个基准上超越现有方法,准确率提升且训练更快
- 适合需要高效学习的GUI自动化系统开发者
图形用户界面(GUI)定位将自然语言指令映射到目标元素的视觉坐标,是自主GUI代理的核心能力。近期强化学习方法(如GRPO)表现优异,但依赖多次采样,且在困难样本上信号稀疏。为解决此问题,本文提出首个面向GUI定位的在线策略自蒸馏框架GUI-SD。首先,利用目标边界框和高斯软掩码构建视觉丰富的教师上下文,提供有效指导而不泄露精确坐标;其次,采用熵引导的蒸馏机制,根据数字重要性和教师置信度自适应加权,聚焦于最具影响且可靠的定位区域。在六个代表性GUI定位基准上的实验表明,GUI-SD在准确率和训练效率上均持续优于基于GRPO的方法及朴素自蒸馏方法。代码与训练数据见https://zhangyan-ucas.github.io/GUI-SD/。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) grounding maps natural language instructions to the visual coordinates of target elements and serves as a core capability for autonomous GUI agents. Recent reinforcement learning methods (e.g., GRPO) have achieved strong performance, but they rely on expensive multiple rollouts and suffer from sparse signals on hard samples. These limitations make on-policy self-distillation (OPSD), which provides dense token-level supervision from a single rollout, a promising alternative. However, its applicability to GUI grounding remains unexplored. In this paper, we present GUI-SD, the first OPSD framework tailored for GUI grounding. First, it constructs a visually enriched privileged context for the teacher using a target bounding box and a Gaussian soft mask, providing informative guidance without leaking exact coordinates. Second, it employs entropy-guided distillation, which adaptively weights tokens based on digit significance and teacher confidence, concentrating optimization on the most impactful and reliable positions. Extensive experiments on six representative GUI grounding benchmarks show that GUI-SD consistently outperforms GRPO-based methods and naive OPSD in both accuracy and training efficiency. Code and training data are available at https://zhangyan-ucas.github.io/GUI-SD/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。