通过分析用户完整操作轨迹,让AI预测下一步行为。
Learning Next Action Predictors from Human-Computer Interaction
- 融合参数化与上下文学习,建模长期交互历史。
- 在20人手机使用数据上预测准确率达17.1%(置信度高时达26%)。
- 适合构建主动式智能助手,尤其关注行为预测场景。
真正的主动型人工智能需预判用户下一步动作。这要求超越输入提示的稀疏信号,而应基于用户所见所行的完整上下文进行推理。本文提出下一动作预测(NAP)任务:给定用户与计算机的多模态交互序列(截图、点击、传感器数据),预测其下一步操作。为解决数据瓶颈,我们利用视觉语言模型标注连续一个月的自然化手机使用数据,构建开源标注流水线,在20名用户的设备上标注超36万次动作,累计1800小时屏幕时间。随后提出LongNAP模型,结合参数化与上下文学习,通过策略梯度训练生成用户专属推理轨迹,从历史轨迹库中检索相关片段,并以检索内容为上下文预测未来动作。采用大模型作为评判标准(0-1相似度),LongNAP在未见数据上相比监督微调和提示基线分别提升79%和39%。此外,跨用户泛化表现良好。尽管用户可能的操作空间庞大(数千种),但仍有17.1%的预测轨迹与真实行为高度一致(评分≥0.5),信心高的预测达到26%。结果表明,基于完整行为上下文进行需求预判已成为可行且潜力巨大的方向。
原文摘要 · Abstract (English)
Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts -- it demands reasoning over the entire context of what we see and do. We formalize this as next action prediction (NAP): given a sequence of a user's multimodal interactions with a computer (screenshots, clicks, sensor data), predict that user's next action. Progress on this task requires both new data and modeling approaches. To scale data, we annotate longitudinal, naturalistic computer use with vision-language models. We release an open-source pipeline for performing this labeling on private infrastructure, and label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time. We then introduce LongNAP, a user model that combines parametric and in-context learning to reason over long interaction histories. LongNAP is trained via policy gradient methods to generate user-specific reasoning traces given some context; retrieve relevant traces from a library of past traces; and then apply retrieved traces in-context to predict future actions. Using an LLM-as-judge evaluation metric (0-1 similarity to ground truth), LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data (by 79% and 39% respectively). Additionally, LongNAP generalizes to held out users when trained across individuals. The space of next actions a user might take at any moment is unbounded, spanning thousands of possible outcomes. Despite this, 17.1% of LongNAP's predicted trajectories are well-aligned with what a user does next (LLM-judge score $\geq$ 0.5). This rises to 26% when we filter to highly confident predictions. In sum, we argue that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。