让手机智能体学会混合使用界面、命令和工具完成可验证的任务。
PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions

- 设计混合操作框架,支持界面、命令行与工具协同执行。
- 在标注数据集上达到75.0%任务通过率,领先旧方法12.9个百分点。
- 适合研究手机自动化、智能体决策与可验证执行的开发者。
手机智能体正被期待完成真实移动工作流,而不仅预测下一屏操作。然而当前多数研究仍将其视为仅依赖屏幕观测并输出点击/滑动的GUI控制器,以目标应用状态评分。真实手机任务更复杂:需判断何时使用应用界面、设备命令或结构化工具,并留下可验证的执行证据。我们提出PhoneHarness,一个支持混合操作的基准与执行框架,可在设备端循环执行GUI、CLI与主机侧工具动作,结合确定性动作路由、有限界面委托与可审计执行日志。其基准PhoneHarness Bench评估智能体是否真正完成任务并产生可观测副作用,而非仅生成看似合理的答案。在标注评估集上,PhoneHarness达75.0%通过率,优于最强非PhoneHarness设置12.9个百分点。因此,PhoneHarness使混合手机工作流可执行,PhoneHarness Bench则衡量智能体使用该框架的可靠性与安全性。结果表明,可靠手机自动化依赖于动作表面路由与可验证执行,而不仅是视觉控制。
原文摘要 · Abstract (English)
Phone agents are increasingly expected to complete real mobile workflows rather than merely predict the next screen action. However, much of the current mobile-agent literature still evaluates agents primarily as GUI controllers that observe a screen, emit taps and swipes, and are scored by target app state. Real phone-use tasks are broader: they require deciding when to use app GUIs, device-side commands, or structured tools, while leaving evidence that the intended side effect actually occurred. We introduce PhoneHarness, a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile workflows. PhoneHarness runs a device-side agent loop over GUI, CLI, and host-side tool actions, combining deterministic action routing with bounded GUI delegation and auditable execution traces. Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers. On the annotated evaluation split, PhoneHarness reaches a 75.0% pass rate, outperforming the strongest non-PhoneHarness settings by 12.9 percentage points. PhoneHarness and PhoneHarness Bench therefore play distinct but mutually dependent roles: the harness makes mixed phone workflows executable, while the benchmark measures whether agents can use that harness reliably and safely. Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。