从屏幕视频自动构建用户行为模型，无需人工标注。
SERUM: State Extraction and Refinement for User Modeling
- 分步迭代标注：先识活动再推意图，用上下文不断优化标签
- 在61段视频上，马尔可夫模型预测准确率显著高于基线，编码任务提升最明显
- 生成的模型结构清晰，人类评估认为比初版标签更准确、有意义
能够主动、个性化交互的智能助手需要结构化的用户意图与工作流模型。然而，从原始非结构化屏幕活动中构建这些模型仍是一大挑战。我们提出SERUM，一种多轮次框架，通过层级视觉语言模型（VLM）标注，直接从非结构化第一人称视频中提取有限状态行为模型。该方法对屏幕记录采用滑动窗口处理，交替执行活动识别与意图推理步骤，每轮利用累积的上下文信息精炼标签，减少单轮标注中的幻觉与时间混淆问题。相似状态通过句向量和人工校准阈值合并为紧凑、连贯的分类体系。我们通过在所得标签序列上拟合一阶马尔可夫模型，并与频率基线对比预测准确率来评估行为结构。在四个领域（编程、烹饪、体力活动、日常生活）共61段第一人称视频上，实验发现：(1) 迭代标注最终收敛至稳定的状态词汇表，称为‘图式平衡’；(2) 归一化马尔可夫模型在预测精度上显著优于频率基线，尤其在结构化任务如编程中表现最佳；(3) 人类标注者评价最终轮标签在准确性与语义表达上均优于首轮。据我们所知，SERUM是首个无需人工标注即可从非结构化第一人称屏幕视频中生成可解释过程模型的系统，为野外用户建模与行为理解提供了可扩展路径。演示、代码与结果均已公开。
原文摘要 · Abstract (English)
Agentic assistants capable of proactive, personalized interactions require structured models of user intent and workflow. However, building these models from raw, unstructured screen activity remains an open challenge. We present SERUM, a multi-pass framework that extracts finite-state behavioral models directly from unstructured egocentric video using hierarchical VLM annotation. Processing screen recordings through a sliding window, SERUM alternates between activity-recognition and intent-inference passes, with each pass refining labels using accumulated prior context to reduce hallucination and temporal conflation seen in single-pass annotation. Synonymous states are then merged via sentence embeddings and human-calibrated thresholds into a compact, coherent taxonomy. We evaluate behavioral structure by fitting first-order Markov models over the resulting label sequences (both actions and intents) and measuring predictive accuracy against frequency baselines. Across 61 egocentric videos in four domains (coding, cooking, physical activities, and daily life), we find: (1) iterative label refinement converges to a stable state vocabulary, which we term schematic equilibrium, after several passes; (2) normalized Markov models achieve substantially lower perplexity and higher action predictions than frequency baselines, with the largest gains on structured tasks like coding; and (3) human annotators rate final-pass labels as accurate and meaningfully improved over first-pass labels. To our knowledge, SERUM is the first system to produce interpretable process models from unstructured egocentric screen video without manual annotation, opening a scalable pathway for user modeling and behavioral understanding in the wild. Our demo, code, and results are publicly available
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。