arXiv:2506.08972cs.CL2025-06被引 11

构建新基准与调度系统,提升移动端智能体处理复杂任务能力

Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System

  • 设计可交互的UI-NEXUS基准,涵盖三类组合操作任务
  • 现有智能体在组合任务上成功率仅24%~40%提升,存在执行不足等问题
  • 提出AGENT-NEXUS调度系统,动态分解长程任务提升效率

基于多模态大语言模型的自主智能体已在移动设备上用于任务执行。然而,以往研究主要关注原子任务(如单屏定位、短链执行),忽视了对组合任务的泛化能力,而后者对真实应用至关重要。本文提出UI-NEXUS,一个全面的基准,用于评估移动端智能体在三类组合操作上的表现:简单拼接、上下文切换和深度探索。该基准支持20个完全可控的本地实用应用环境及30个在线中英文服务应用,包含100个交互式任务模板,平均最优步数为14.05。实验表明,现有智能体在组合任务上普遍面临性能与效率的权衡,出现执行不足、过度执行和注意力漂移等典型失败模式,暴露显著的原子到组合泛化差距。受此启发,本文提出AGENT-NEXUS,一种轻量高效的调度系统,通过动态将长程任务分解为一系列自洽的原子子任务,扩展现有智能体能力。在不显著增加推理开销的前提下,使现有智能体在组合操作任务上的成功率提升24%至40%。演示视频、数据集与代码已公开于项目主页 https://ui-nexus.github.io。

原文摘要 · Abstract (English)

Autonomous agents powered by multimodal large language models have been developed to facilitate task execution on mobile devices. However, prior work has predominantly focused on atomic tasks -- such as shot-chain execution tasks and single-screen grounding tasks -- while overlooking the generalization to compositional tasks, which are indispensable for real-world applications. This work introduces UI-NEXUS, a comprehensive benchmark designed to evaluate mobile agents on three categories of compositional operations: Simple Concatenation, Context Transition, and Deep Dive. UI-NEXUS supports interactive evaluation in 20 fully controllable local utility app environments, as well as 30 online Chinese and English service apps. It comprises 100 interactive task templates with an average optimal step count of 14.05. Experimental results across a range of mobile agents with agentic workflow or agent-as-a-model show that UI-NEXUS presents significant challenges. Specifically, existing agents generally struggle to balance performance and efficiency, exhibiting representative failure modes such as under-execution, over-execution, and attention drift, causing visible atomic-to-compositional generalization gap. Inspired by these findings, we propose AGENT-NEXUS, a lightweight and efficient scheduling system to tackle compositional mobile tasks. AGENT-NEXUS extrapolates the abilities of existing mobile agents by dynamically decomposing long-horizon tasks to a series of self-contained atomic subtasks. AGENT-NEXUS achieves 24% to 40% task success rate improvement for existing mobile agents on compositional operation tasks within the UI-NEXUS benchmark without significantly sacrificing inference overhead. The demo video, dataset, and code are available on the project page at https://ui-nexus.github.io.

移动智能体组合泛化任务调度基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。