用双层机制让少量专家轨迹提升端到端GUI智能体表现
From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation
- 通过自洽轨迹生成和动态缓存，将专家数据适配为策略引导
- 在OSWorld-Verified上成功率从22.87%提升至32.13%
- 适合研究端到端视觉语言智能体与强化学习的开发者
视觉语言模型正被广泛用于操作桌面和浏览器的计算机使用代理（CUAs）。顶尖性能的CUAs采用框架式系统，将规划与执行分离，而端到端的截图到动作策略部署更简单，但在OSWorld-Verified等基准上表现较差。GUI数据集如OSWorld存在双重瓶颈：仅包含数百个可交互、可验证的任务与环境，且专家轨迹需通过与环境交互获取，难以扩展。因此我们探讨如何利用少量专家轨迹，通过可验证奖励的强化学习（RLVR）训练端到端策略。直接将离策略轨迹混入在线策略训练易失败：即使格式转换后，专家轨迹仍存在结构不匹配和分布偏移。为此提出BEPA（双层专家到策略同化），通过基础策略生成自滚动可达轨迹（LEVEL-1）和在RLVR中动态更新的任务级缓存（LEVEL-2），实现专家数据到策略对齐的转化。在OSWorld-Verified上，BEPA将UITARS1.5-7B的成功率从22.87%提升至32.13%，持留测试集从5.74%提升至10.30%，并在MMBench-GUI和Online-Mind2Web上保持一致提升。代码与数据见：https://github.com/LEON-gittech/Verl_GUI.git
原文摘要 · Abstract (English)
Vision-language models are increasingly deployed as computer-use agents (CUAs) that operate desktops and browsers. Top-performing CUAs are framework-based systems that decompose planning and execution, while end-to-end screenshot-to-action policies are easier to deploy but lag behind on benchmarks such as OSWorld-Verified. GUI datasets like OSWorld pose two bottlenecks: they expose only a few hundred interactive, verifiable tasks and environments, and expert trajectories must be gathered by interacting with these environments, making such data hard to scale. We therefore ask how reinforcement learning from verifiable rewards (RLVR) can best exploit a small pool of exist expert trajectories to train end-to-end policies. Naively mixing these off-policy traces into on-policy RLVR is brittle: even after format conversion, expert trajectories exhibit structural mismatch and distribution shift from the learner. We propose BEPA (Bi-Level Expert-to-Policy Assimilation), which turns static expert traces into policy-aligned guidance via self-rolled reachable trajectories under the base policy (LEVEL-1) and a per-task, dynamically updated cache used in RLVR (LEVEL-2). On OSWorld-Verified, BEPA improves UITARS1.5-7B success from 22.87% to 32.13% and raises a held-out split from 5.74% to 10.30%, with consistent gains on MMBench-GUI and Online-Mind2Web. Our code and data are available at: https://github.com/LEON-gittech/Verl_GUI.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。