通过技能引导生成新示范，解决GUI代理在偏离专家轨迹时的监督空白问题。
Skill-Guided Continuation Distillation for GUI Agents
- 用无技能引导策略先到达真实偏离轨迹状态，再由技能引导策略完成任务并生成示范。
- 在OSWorld-Verified上将三款基础模型成功率从30%提升至50%以上。
- 适用于需长期执行且易偏离专家路径的GUI自动化任务，如界面操作代理。
提升GUI代理通常依赖专家轨迹的行为克隆。然而，当当前策略偏离专家策略时，在闭环执行中不可避免地会遇到政策诱导的非轨迹状态，即超出专家轨迹范围的状态。由于专家轨迹未提供这些未见状态的示范，此类状态缺乏有效监督，导致策略无法选择正确动作。为填补这一监督空白，我们提出技能引导延续蒸馏（SGCD），一种迭代自提升框架。SGCD首先在无技能引导下运行原始策略几步，以达到真实的非轨迹状态；随后，由技能引导策略从这些状态继续完成任务，并生成成功延续，与专家轨迹混合，为政策诱导的非轨迹状态提供监督。技能从成功与失败的滚动中提取，包括延续计划、关键目标、失败陷阱和成功标准。在OSWorld-Verified上，SGCD将三种基础模型的成功率从低30%范围提升至50%以上，证明了其有效性与通用性。
原文摘要 · Abstract (English)
Improving GUI agents typically relies on behavior cloning on expert trajectories. However, as the current policy deviates from the expert policy, it inevitably encounters policy-induced off-trajectory states during closed-loop execution, i.e., states that fall outside the expert trajectories. Since expert trajectories provide no demonstrations for these unseen states, such states receive no effective supervision, leaving the policy unable to select the correct action. To close this supervision gap, we propose Skill-Guided Continuation Distillation (SGCD), an iterative self-improvement framework. SGCD first runs the plain policy without skill guidance for a few steps to reach realistic off-trajectory states. From these states, a skill-guided policy then completes the task and produces successful continuations, which are mixed with expert trajectories to supply supervision over policy-induced off-trajectory states. The skills are extracted from both successful and failed rollouts, consisting of Continuation Plans, Critical Targets, Failure Traps, and Success Criteria. On OSWorld-Verified, SGCD improves the success rate of three base models from the low-30\% range to over 50\%, demonstrating its effectiveness and generality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。