轻量级模型实现手机应用精准控制，实时高效。
Lightweight Neural App Control
- 用文本目标+历史界面信息生成动作，结合小模型实现实时决策。
- 相比微调版VLM提升19%动作准确率，比提示工程方案高42%。
- 适合资源受限设备，对移动智能助手有实际应用价值。
本文提出一种新型移动端控制架构——轻量级多模态应用控制（LiMAC），用于在各类Android应用中实现高效交互与控制。LiMAC以文本目标和一系列过往移动观测（如截图与对应的UI树）为输入，生成精确操作指令。为应对智能手机的计算限制，我们引入一个小型动作变换器（AcT），并结合微调后的视觉语言模型（VLM）实现实时决策与任务执行。我们在两个开源移动端控制数据集上评估了LiMAC，结果表明该小型化方案在性能上优于微调后的开源VLM（如Florence2和Qwen2-VL），且显著超越使用闭源基础模型（如GPT-4o）的提示工程基线。具体而言，相较于微调版VLM，LiMAC将整体动作准确率提升最高达19%；相较于提示工程基线，提升最高达42%。
原文摘要 · Abstract (English)
This paper introduces a novel mobile phone control architecture, Lightweight Multi-modal App Control (LiMAC), for efficient interactions and control across various Android apps. LiMAC takes as input a textual goal and a sequence of past mobile observations, such as screenshots and corresponding UI trees, to generate precise actions. To address the computational constraints inherent to smartphones, we introduce a small Action Transformer (AcT) integrated with a fine-tuned vision-language model (VLM) for real-time decision-making and task execution. We evaluate LiMAC on two open-source mobile control datasets, demonstrating the superior performance of our small-form-factor approach against fine-tuned versions of open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly outperforms prompt engineering baselines utilising closed-source foundation models like GPT-4o. More specifically, LiMAC increases the overall action accuracy by up to 19% compared to fine-tuned VLMs, and up to 42% compared to prompt-engineering baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。