通过难易度感知强化学习,提升图形界面定位的精度与效率。
GUI-C$^2$: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning

- 按样本难易度筛选并加权训练数据,提升训练效率。
- 采用粗到精渐进式区域裁剪,平衡上下文与冗余,准确率显著提升。
- 简化决策流程,降低推理开销,适合轻量级模型部署。
现有面向图形界面定位的智能体强化学习方法在数据与策略层面存在局限。数据层面,当前方法对所有训练样本一视同仁,但其对基础模型的训练价值随难度变化,忽视此差异会大幅降低训练效率甚至导致训练崩溃。策略层面,现有框架难以平衡扩大裁剪区域以获取充分上下文与缩小区域以减少冗余之间的矛盾,且复杂决策机制不适用于小参数模型,显著增加推理时间。为此,我们提出GUI-D,一个数据挖掘与难易度评分流程,通过有效测试识别高价值样本并分配难易度分数以指导后续训练权重。同时提出GUI-C²,采用区域门控的粗到精细化机制,利用模型内部不确定性信号逐步缩小视觉范围,自适应保留大目标上下文并增强小目标精度,结合改进感知阶段奖励确保每轮精修均带来真实性能提升。此外,简化决策过程以大幅降低额外推理时间。大量实验表明,本方法达到当前最优性能。代码与数据将公开。
原文摘要 · Abstract (English)
Existing agentic reinforcement learning methods for GUI grounding have limitations at two levels. At the data level, current approaches typically treat all training samples equally, although their training value to the baseline model varies with difficulty. Overlooking this can greatly reduce training efficiency or even cause collapse. At the strategy level, existing frameworks struggle to balance the trade-off between cropping larger regions for sufficient context and smaller ones for reduced redundancy, a tension inherent to tool-augmented grounding agents. In addition, overly complex decision-making is difficult for small-parameter models and significantly increases inference time. To address these issues, at the data level, we propose GUI-D, a data mining and difficulty scoring pipeline that identifies the training-worthy samples by proper testing and assigns difficulty scores to guide subsequent training weights. At the strategy level, we propose GUI-C$^2$, which employs an area-gated coarse-to-fine refinement mechanism that progressively narrows the visual field via model-internal uncertainty signals, adaptively reserving context for large targets while amplifying precision for small ones, reinforced by improvement-aware stage rewards that ensure each refinement genuinely advances grounding. Meanwhile, we simplify the decision-making process to greatly reduce additional inference time. Finally, extensive experiments show that our method achieves state-of-the-art performance. The code and data will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。