用人类规划的子任务提升手机助手执行效率
CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planning
- 以人工规划的子任务为基准向量，优化GUI场景下的任务分解
- 在20个应用中实现效果与效率双重提升，中英文均有效
- 适合需要高可靠性的移动端智能助手开发者参考
视觉语言模型（VLM）的发展使手机操作助手能模拟人类行为完成用户需求。现有基于VLM的手机助手分为任务、子任务和动作三个层级，其中连接高层目标与底层可执行动作的子任务层级至关重要，但面临两大挑战：无效子任务导致下层代理无法执行，低效子任务未能推进高层任务完成。根源在于VLM在多智能体架构下缺乏GUI场景中的子任务分解经验。为此，我们提出约束高频优化子任务规划（CHOP）的新架构。通过以人类规划的子任务作为基向量，克服VLM在GUI场景规划中的不足。我们在中英文环境下对20个应用进行了评估，显著提升了任务完成的效果与效率。数据集与代码已公开于https://github.com/Yuqi-Zhou/CHOP。
原文摘要 · Abstract (English)
The advancement of visual language models (VLMs) has enhanced mobile device operations, allowing simulated human-like actions to address user requirements. Current VLM-based mobile operating assistants can be structured into three levels: task, subtask, and action. The subtask level, linking high-level goals with low-level executable actions, is crucial for task completion but faces two challenges: ineffective subtasks that lower-level agent cannot execute and inefficient subtasks that fail to contribute to the completion of the higher-level task. These challenges stem from VLM's lack of experience in decomposing subtasks within GUI scenarios in multi-agent architecture. To address these, we propose a new mobile assistant architecture with constrained high-frequency o}ptimized planning (CHOP). Our approach overcomes the VLM's deficiency in GUI scenarios planning by using human-planned subtasks as the basis vector. We evaluate our architecture in both English and Chinese contexts across 20 Apps, demonstrating significant improvements in both effectiveness and efficiency. Our dataset and code is available at https://github.com/Yuqi-Zhou/CHOP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。