用真实手机操作数据自动生成可复现的手机智能体环境
PhoneWorld: Scaling Phone-Use Agent Environments

- 从真实手机界面轨迹中自动提取关键屏幕和交互逻辑
- 在固定训练预算下,使4个评测基准平均提升超10分
- 适合研究手机智能体泛化与大规模环境构建的学者
手机智能体发展的瓶颈在于难以规模化构建可控、可复现的真实移动行为环境。现有基准虽推动了评估进展,但无法高效生成新环境。本文提出PhoneWorld,一个可复用的流水线,将真实GUI轨迹与截图转化为可执行的手机环境、任务、自动验证器及训练数据。它通过分析真实轨迹,自动识别关键界面、状态变化点与可验证目标,构建基于只读应用内容与可变状态的模拟安卓应用。当前版本覆盖16个领域34个应用,涵盖搜索、浏览、购物、预订、媒体与社交等常见行为。在固定训练预算下,用PhoneWorld替代基线中的1万步辅助数据,使HYMobileBench提升17.7分,AndroidControl提升6.0分,AndroidWorld提升14.7分,PhoneWorld自身提升52.5分。进一步研究表明:增加监督数据量显著提升性能,且在固定预算下扩展应用覆盖范围带来更大收益。总体上,PhoneWorld实现了从逐个构建基准到规模化供给环境的范式转变。
原文摘要 · Abstract (English)
A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale. Existing mobile-agent benchmarks have made important progress on evaluation, but they do not by themselves provide a scalable way to construct many new phone-use environments. We present PhoneWorld, a reusable pipeline that converts real GUI trajectories and screenshots into controllable phone-use environments, executable tasks, automatic verifiers, and training rollouts. Rather than hand-building one mobile benchmark at a time, PhoneWorld uses real trajectories to recover which screens matter, how screens connect, which interactions must change environment state, and which user goals admit automatic verification. From these signals, it builds runnable mock Android apps backed by read-only app content and mutable state, then derives executable tasks, rule-based verifiers, and training rollouts from the same environments. In its current instantiation, PhoneWorld covers 34 apps across 16 domains, spanning common consumer mobile behaviors such as search, browsing, shopping, booking, media, and social interaction. Under a fixed training budget, replacing 10K steps from an auxiliary AndroidWorld corpus in an AndroidWorld-based baseline with broad PhoneWorld supervision improves all four evaluation benchmarks at once, raising HYMobileBench by 17.7 points, AndroidControl by 6.0 points, AndroidWorld by 14.7 points, and PhoneWorld by 52.5 points. We then study two additional scaling questions: increasing the amount of PhoneWorld supervision strongly improves PhoneWorld performance, and under a fixed PhoneWorld budget, expanding app coverage yields even larger gains. Overall, PhoneWorld shifts the focus from building one mobile benchmark at a time to scaling the supply of phone-use environments themselves.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。