评测手机端智能体在真实任务中的工具调用与规划能力
MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks
- 构建可交互的沙箱环境，模拟真实手机运行状态
- 覆盖13个功能领域212个实用工具，评估复杂任务表现
- 聚焦协作、记忆、技能复用三大核心能力，适合研究移动智能体者
随着本地大模型智能体演变为个人协作者，移动端成为该范式的关键测试平台，严谨的能力评估至关重要。现有基准存在两大盲区：以图形界面为中心的基准仅测试表面操作，忽略后台工具使用和长程规划；静态函数调用基准依赖离线API匹配，脱离实际运行约束。为此，我们提出 extbf{MobilePA-Bench}，一个交互式、状态化、以工具为核心的基准，用于评估移动端规划智能体的工具调用与规划能力。该基准运行于保持实时应用数据库并返回结构化反馈的可执行沙箱中，涵盖13个功能领域和212个真实手机工具。除基础工具使用外，还从三个维度评估智能体：(1) 子智能体协作——分解复杂任务并委派给专业子智能体；(2) 记忆使用——调用存储的记忆、用户档案和过往偏好以理解隐含请求；(3) 技能使用——调用预打包的复合技能而非从零规划每一步。大量实验表明，当前前沿大模型在移动端仍不可靠：在严格工具顺序、权限限制及意外运行时错误下性能急剧下降。通过结合交互式函数调用沙箱与证据验证，MobilePA-Bench既可用作实践诊断基准，也可作为智能体强化学习的交互基础，加速可靠移动端智能体的发展。
原文摘要 · Abstract (English)
As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present \textbf{MobilePA-Bench}, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning $13$ functional domains and $212$ realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: \emph{(1)~Sub-agent Collaboration}---decomposing a complex task and delegating specialized work to capable sub-agents; \emph{(2)~Memory Usage}---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and \emph{(3)~Skill Usage}---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。