用小模型生成代码实现手机自动化,本地运行更安全高效。
AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation
- 将手机操作转为代码生成任务,由本地小模型执行
- 在多个应用上达成90%以上成功率,延迟更低、消耗更少
- 适合注重隐私、需低延迟的移动端自动化场景
大语言模型(LLMs)推动了移动UI代理的发展,但现有系统依赖强大模型,难以在终端设备本地部署,引发用户隐私与中心化服务成本问题。受近期小语言模型(SLMs)强大编码能力启发,我们提出将移动任务自动化转化为代码生成问题,由本地SLM与代码解释器协同完成。由于目标应用多样复杂,无法依赖公开数据集预训练,因此采用文档中心方法,为每个应用自动生成细粒度API文档,并基于此生成多样化任务样本。通过合成文档与样本引导,模型学会生成精确高效的脚本以完成未见任务。与最先进移动代理相比,该方法显著提升任务成功率,降低延迟和每任务令牌消耗。代码已开源:https://github.com/MobileLLM/AutoDroid-V2。
原文摘要 · Abstract (English)
Large language models (LLMs) have brought exciting new advances to mobile UI agents, a long-standing research field that aims to complete arbitrary natural language tasks through mobile UI interactions. However, existing UI agents usually demand powerful large language models that are difficult to be deployed locally on end-users' devices, raising huge concerns about user privacy and centralized serving cost. Inspired by the remarkable coding abilities of recent small language models (SLMs), we propose to convert the UI task automation problem to a code generation problem, which can be effectively solved by an on-device SLM and efficiently executed with an on-device code interpreter. Unlike normal coding tasks that can be extensively pre-trained with public datasets, generating UI automation code is challenging due to the diversity, complexity, and variability of target apps. Therefore, we adopt a document-centered approach that automatically builds fine-grained API documentation for each app and generates diverse task samples based on this documentation. By guiding the agent with the synthetic documents and task samples, it learns to generate precise and efficient scripts to complete unseen tasks. Based on detailed comparisons with state-of-the-art mobile UI agents, our approach effectively improves the mobile task automation with significantly higher success rates and lower latency/token consumption. Code is open-sourced at https://github.com/MobileLLM/AutoDroid-V2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。