arXiv:2605.28534cs.CL2026-05被引 1

让AI更懂图形界面操作,提升任务完成率

GUI-CIDER: Mid-training GUI Agents via Causal Internalization and Density-aware Exemplar Reselection

论文配图:GUI-CIDER: Mid-training GUI Agents via Causal Internalization and Density-aware Exemplar Reselection
图 1 · 摘自论文原文
  • 通过因果内化与密度感知重选,显式学习界面操作知识
  • 在两个知识基准上任务成功率提升12%-18%
  • 适合需要精准界面理解的自动化工具开发者

尽管多模态大模型在构建图形用户界面(GUI)代理方面取得快速进展,但其在真实场景中的任务完成能力仍受限于对界面操作的世界知识缺乏。现有方法通常依赖昂贵的多代理框架或传统的后训练范式(如监督微调SFT和强化学习RL),但这些方法仅能通过动作标注或奖励信号隐式吸收知识,导致轨迹记忆效率低,而非真正理解。为此,我们提出GUI-CIDER,一种中段训练方法,通过因果内化与密度感知示例重选,显式内化GUI世界知识。该方法分为三阶段:(1) 数据合成,从GUI轨迹中提炼静态规划与动态因果知识并转化为文本;(2) 示例重选,基于因果结构奖励与语义冗余惩罚筛选语料;(3) 中段训练,利用优化数据嵌入所获知识。在两个GUI知识基准和三个任务完成基准上的实验表明,GUI-CIDER持续提升了代理对界面操作的理解能力与任务成功率。

原文摘要 · Abstract (English)

Despite the rapid progress of multimodal large language models in building Graphical User Interface (GUI) agents, their real-world task completion is fundamentally bottlenecked by a lack of world knowledge about GUI operations. Existing solutions typically rely on expensive multi-agent scaffolding or conventional post-training paradigms, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). However, post-training only allows agents to implicitly absorb world knowledge through action annotations or reward signals, leading to inefficient trajectory memorization rather than genuine comprehension. Therefore, an approach that enables explicit learning of this knowledge is imperative. To this end, we propose GUI-CIDER, a mid-training method that explicitly internalizes GUI world knowledge through Causal Internalization and Density-aware Exemplar Reselection. GUI-CIDER operates in three stages: (1) data synthesis, which distills static planning and dynamic causal knowledge from GUI trajectories into text; (2) exemplar reselection, which filters the corpus by rewarding causal structures and penalizing semantic redundancy; and (3) mid-training, where the refined data is used to embed the acquired knowledge. Extensive experiments on two GUI knowledge benchmarks and three task completion benchmarks demonstrate that GUI-CIDER consistently improves both the agent's understanding of GUI operations and its task success rates.The codes are available at https://github.com/Wuzheng02/GUI-CIDER.

GUI代理因果学习知识内化任务完成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。