用多教师在线蒸馏,让一个GUI代理同时掌握电脑和手机操作
UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents

- 通过动态路由让学生代理在不同平台调用专用教师模型
- 跨平台任务成功率分别达38.2%和12.0%,优于融合模型
- 适合需要统一控制多设备的自动化系统开发者
近年来,多模态基础模型与智能体系统的发展推动了GUI代理从单平台任务执行向跨平台交互演进。然而,统一的多平台GUI学习仍面临挑战:高质量跨平台轨迹稀缺,各平台虽有可迁移能力,但动作语义与交互规范差异显著。直接混合监督信号或合并专用模型会模糊原生行为,导致性能失衡。为此,我们构建了Uni-GUI数据集,包含近10,000条通过统一桌面-移动端框架收集的可执行跨平台交互轨迹。基于此,提出首个将多教师在线蒸馏(MOPD)引入统一多平台GUI代理训练的UI-MOPD框架。该方法在学生代理自生成的回放缓冲区中,动态将其每条轨迹路由至对应平台专用教师。在学生访问的状态下,教师指导作为平台条件的行为锚点,实现桌面与移动端技能互补,而无需平均其差异化的交互惯例。在OSWorld和MobileWorld测试中,UI-MOPD分别取得38.2%和12.0%的任务成功率,超越参数匹配的集成策略,同时保持通用GUI理解能力。结果表明,多教师在线蒸馏为构建统一跨平台GUI代理提供了有效路径。
原文摘要 · Abstract (English)
Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction. However, unified multi-platform GUI learning remains challenging: high-quality cross-platform trajectories remain scarce, while platforms share transferable capabilities but differ in action semantics and interaction conventions. Naively mixing supervision or merging specialized models can blur native behaviors and produce imbalanced performance. To address these challenges, we construct Uni-GUI, a high-quality dataset containing nearly 10K executable cross-platform interaction trajectories collected through a unified desktop-mobile harness. Building on Uni-GUI, we propose UI-MOPD, the first framework to introduce multi-teacher on-policy distillation (MOPD) into unified multi-platform GUI agent training. UI-MOPD trains a shared student on its own rollouts and dynamically routes each rollout to the corresponding platform-specialized teacher. At student-visited states, teacher guidance serves as a platform-conditioned behavioral anchor, enabling the integration of complementary desktop and mobile expertise without averaging their distinct interaction conventions. On OSWorld and MobileWorld, UI-MOPD achieves task success rates of 38.2% and 12.0%, respectively, outperforming parameter-matched integration strategies while preserving general GUI grounding. These results demonstrate that multi-teacher on-policy distillation provides an effective approach to building unified cross-platform GUI agents. Project page: https://elispectre.github.io/UI-MOPD/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。