构建2万条真实手机操作数据集，推动智能助手主动预判用户需求。
FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents
- 基于2万条真实安卓操作记录，融合位置、时间等上下文信息
- 提出主动推荐与个性化执行双赛道，提升任务适配度
- 实验证明现有模型与人类表现差距大，数据可显著提升智能体能力
移动GUI智能体正成为提升智能设备体验的关键工具，多模态大语言模型（MLLM）已成为该领域的主流范式。然而，当前智能体仍依赖显式指令，忽视利用位置、时间、用户画像等上下文信息及历史数据进行主动任务建议。此外，以往研究侧重任务执行成功率，却较少关注个性化执行路径，忽略了用户偏好差异。为此，我们提出了FingerTip 20K基准。收集了20,000条跨多种日常应用的多步安卓交互人类示范，数据源自用户长期真实使用，包含关键用户相关上下文信息。基准包含两个新赛道：基于环境观察和历史意图的主动任务建议，以及针对用户动作偏好的个性化任务执行。实验表明，这些新赛道对利用用户信息处理GUI任务构成重大挑战。人类研究表明，现有智能体与人类表现存在巨大差距。使用本数据微调的模型能有效利用用户信息并取得良好效果，凸显该方法在构建更以人为中心的移动LLM智能体中的潜力。代码已开源：https://github.com/tsinghua-fib-lab/FingerTip-20K。
原文摘要 · Abstract (English)
Mobile GUI agents are becoming critical tools to improve user experience on smart devices, with multimodal large language models (MLLMs) emerging as the dominant paradigms in this domain. Current agents, however, rely on explicit human instructions, overlooking the potential to leverage the contextual information (like location, time, user profile) and historical data for proactive task suggestions. Besides, previous works focus on optimizing the success rate during task execution, but pay less attention to the personalized execution trajectory, thereby neglecting potentially vast differences in user preferences. To address these challenges, we introduce the FingerTip 20K benchmark. We collected 20K unique human demonstrations of multi-step Android device interactions across a variety of everyday apps. These demonstrations are not isolated but are continuously acquired from the users' long-term usage in their real lives, and encompass essential user-related contextual information. The benchmark contains two new tracks: proactive task suggestions by analyzing environment observation and users' previous intents, and personalized task execution by catering to users' action preferences. Our experiments reveal that the tracks we propose pose significant challenges for leveraging user-related information in GUI tasks. We also performed a human study to show that there exists a huge gap between existing agents and humans. The model fine-tuned with the data we collected effectively utilized user information and achieved good results, highlighting the potential of our approach in building more user-oriented mobile LLM agents. Our code is open-source at https://github.com/tsinghua-fib-lab/FingerTip-20K for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。