让跨平台的GUI智能体学会自我融合，提升任务成功率。
MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation
- 按动作结构重分配训练信号，聚焦错误动作学习。
- 在80亿模型上比最强基线高2.0%成功率，接近专家水平。
- 适合需要跨平台部署的GUI智能体研发人员使用。
基于大语言模型的图形用户界面（GUI）智能体正广泛应用于移动、网页和桌面环境。然而，现有智能体通常局限于特定领域，限制了部署范围与用户体验。这促使将多个专用模型整合为统一的跨环境策略。直接合并权重会因专家意见冲突导致可执行动作损坏，而传统在线策略蒸馏（OPD）虽避免矛盾监督，却对所有输出标记同等对待，忽略了只有动作标记才是智能体与环境交互的唯一接口。为此，本文提出MAGA，根据生成动作的正确性重新分配训练信号：抑制无效或多余蒸馏信号，集中优化错误动作。此外，引入仅训练时使用的提示机制，在不改变学生输入的前提下优化教师提供的监督信号。在两种模型规模下，MAGA均达到最高平均成功率，80亿参数模型上比最强基线高出2.0%，且接近各领域教师的平均性能。
原文摘要 · Abstract (English)
Graphical user interface (GUI) agents based on large language models are increasingly deployed across mobile, web, and desktop environments. However, existing agents are typically domain-specific, limiting the deployment and user experience. This motivates the consolidation of specialized models into a single cross-environment policy. Weight merging directly merges domain-specific experts but can corrupt executable actions under expert disagreement, while on-policy distillation (OPD) avoids conflicting teacher supervision yet still treats all response tokens equally during distillation, ignoring that action tokens are the only interface between the environment and the agent. To address this, We introduce MAGA that re-allocates training signal according to the structured action. Based on the correctness of the generated action, it suppresses unnecessary or invalid distillation signals and focuses learning on erroneous actions. Besides, a training-only hint optimizes the supervision signal provided by domain-specific teachers without changing the student input. Across two model scales, MAGA achieves the highest mean success rate, outperforming the strongest baseline by 2.0% at 8B and achieves almost the same average performance with teachers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。