轻量级视觉语言模型，让手机助手更高效精准执行指令
AppVLM: A Lightweight Vision Language Model for Online App Control
- 用轻量VLM模型+离线+在线数据迭代训练，兼顾效率与泛化能力
- 离线任务准确率最高，线上任务成功率媲美GPT-4o，速度提升十倍
- 适合资源受限场景的手机智能助手部署，尤其关注实时性与成本
将基础模型作为智能手机助手（即应用代理）是当前关键研究挑战。这些代理需根据文本指令理解并操作手机界面完成任务。现有方法存在显著局限：使用大型专有模型（如GPT-4o）计算开销大，而小型微调模型又难以适应分布外任务。本文提出AppVLM，一种轻量级视觉语言模型（VLM）。首先在AndroidControl数据集上离线微调；随后在AndroidWorld环境中收集数据并进行策略优化训练。实验表明，AppVLM在AndroidControl数据集上的离线动作预测准确率高于所有基线，且在AndroidWorld环境中的在线任务完成成功率与GPT-4o相当，但推理速度最快可达其十倍。该模型为实际部署提供了高效可行的解决方案。
原文摘要 · Abstract (English)
The utilisation of foundation models as smartphone assistants, termed app agents, is a critical research challenge. These agents aim to execute human instructions on smartphones by interpreting textual instructions and performing actions via the device's interface. While promising, current approaches face significant limitations. Methods that use large proprietary models, such as GPT-4o, are computationally expensive, while those that use smaller fine-tuned models often lack adaptability to out-of-distribution tasks. In this work, we introduce AppVLM, a lightweight Vision-Language Model (VLM). First, we fine-tune it offline on the AndroidControl dataset. Then, we refine its policy by collecting data from the AndroidWorld environment and performing further training iterations. Our results indicate that AppVLM achieves the highest action prediction accuracy in offline evaluation on the AndroidControl dataset, compared to all evaluated baselines, and matches GPT-4o in online task completion success rate in the AndroidWorld environment, while being up to ten times faster. This makes AppVLM a practical and efficient solution for real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。