让GUI模型部署后自我进化,自动修复错误并提升准确率。
Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

- 通过探索-评估-反思-内化闭环,实现部署后自适应。
- 在六大数据集上平均准确率提升7.4%。
- 适合需要长期维护的自动化GUI系统开发者。
GUI视觉定位是GUI智能体的基础能力。现有模型部署后参数固定,难以适应新界面。尽管近期方法尝试通过测试时强化学习进行适应,但无法反思失败探索。为此,我们提出测试时自演化框架,使模型在无人工标注的情况下持续改进。该框架构建了探索、评估、反思与内化的闭环:代理首先根据指令预测定位坐标;为评估结果,引入基于多模态大模型的反射器,生成推理反思;为将反思知识融入模型权重,提出反射引导的策略内自蒸馏,通过条件自教师将高层推理转化为密集的词元级监督;此外,设计对比校准方法,防止错误自回归前缀污染监督信号。在六个基准上的实验证明,本框架显著有效,平均准确率相比基线提升7.4%。据我们所知,这是首个成功在测试时自适应中应用策略内自蒸馏的GUI视觉定位工作。该框架填补了部署后适应的空白,完整了GUI智能体的自演化能力。代码将公开。
原文摘要 · Abstract (English)
GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, we propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth. It constructs a closed-loop of Exploration, Evaluation, Reflection, and Internalization. Specifically, the agent first explores unseen interfaces by predicting grounding coordinates for given instructions. To evaluate these explorations, we introduce an MLLM-based Reflector to assess the generated results and provide the corresponding reasoning reflections. To internalize reflection knowledge into the model weights, we propose Reflection-Guided On-Policy Self-Distillation, which translates high-level reasoning into dense token-level supervision via a conditioned self-teacher. Furthermore, we design a Contrastive Calibration method to prevent incorrect auto-regressive prefixes from corrupting the supervisory signals during failed explorations. Extensive experiments across six benchmarks demonstrate our framework's effectiveness, achieving an average accuracy improvement of 7.4% over the base model. To the best of our knowledge, this is the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding. By filling the gap in post-deployment adaptation, our framework completes the self-evolving capability of GUI agents. The code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。