让训练难度自动匹配模型能力,提升移动端智能体学习效果。
Learning with Challenges: Adaptive Difficulty-Aware Data Generation for Mobile GUI Agent Training
- 将任务难度分为结构与语义两类,动态追踪模型能力边界。
- 生成数据使智能体性能平均提升1.57倍,跨多个基准测试一致领先。
- 适合想提升移动端智能体训练效率的研究者和开发者。
大规模高质量交互轨迹对推进移动端图形用户界面(GUI)智能体至关重要。现有方法通常依赖人工示范或自动化模型探索生成轨迹,但缺乏对任务难度的细粒度控制,导致训练难度与智能体能力不匹配,限制学习效果。受人类通过逐步挑战任务习得技能的启发,我们提出 MobileGen,一种新型数据生成框架,可自适应地将训练难度与 GUI 智能体的能力前沿对齐。MobileGen 显式将任务难度解耦为结构维度(如轨迹长度)和语义维度(如任务目标),并通过在精选先验数据集上迭代评估智能体,系统构建其在两个维度上的能力轮廓。基于此轮廓,自适应计算任务难度的概率分布,并从中采样下一阶段训练的目标难度。随后,利用多智能体可控生成器合成高质量交互轨迹及对应任务指令。大量实验表明,MobileGen 在多个挑战性基准上均显著优于现有方法,使智能体平均性能提升 1.57 倍,凸显了能力对齐数据生成在有效训练中的重要性。
原文摘要 · Abstract (English)
Large-scale, high-quality interaction trajectories are essential for advancing mobile Graphical User Interface (GUI) agents. While existing methods typically rely on labor-intensive human demonstrations or automated model exploration to generate GUI trajectories, they lack fine-grained control over task difficulty. This fundamentally restricts learning effectiveness due to the mismatch between the training difficulty and the agent's capabilities. Inspired by how humans acquire skills through progressively challenging tasks, we propose MobileGen, a novel data generation framework that adaptively aligns training difficulty with the GUI agent's capability frontier. Specifically, MobileGen explicitly decouples task difficulty into structural (e.g., trajectory length) and semantic (e.g., task goal) dimensions. It then iteratively evaluates the agent on a curated prior dataset to construct a systematic profile of its capability frontier across these two dimensions. With this profile, the probability distribution of task difficulty is adaptively computed, from which the target difficulty for the next round of training can be sampled. Guided by the sampled difficulty, a multi-agent controllable generator is finally used to synthesize high-quality interaction trajectories along with corresponding task instructions. Extensive experiments show that MobileGen consistently outperforms existing data generation methods by improving the average performance of GUI agents by 1.57 times across multiple challenging benchmarks. This highlights the importance of capability-aligned data generation for effective mobile GUI agent training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。