从电脑操作记录中自动提炼出多任务的结构化工作模型。
Inducing Task Models from Computer-Use Traces

- 通过分解并发任务,构建目标递归分解与执行流程结合的模型。
- 在真实轨迹上恢复交错任务匹配度达0.974,还原74.9%的操作步骤。
- 提升新任务准确率30.0%,适合自动化工作代理与知识复用场景。
自然状态下的电脑使用痕迹(如被动记录的截图、鼠标键盘操作)是提取日常工作中可符号化、可审计、可复用模型的宝贵资源。随着计算机代理进入实际工作,理解任务真实执行方式、组织审计和重用知识变得至关重要。然而,现有方法受限于预设任务或单一流程,仅生成步骤级摘要,难以处理多线程交错任务。本文提出任务模型诱导(TMI),能(i)在无约束轨迹中发现隐含任务并分离并发活动;(ii)为每个隐含任务构建包含层级目标分解与控制流过程的结构化模型。内在评估显示,在人工与代理轨迹上,TMI对交错任务的分组匹配度达0.974,还原74.9%的执行步骤,显著优于最强基线。外在评估中,基于TMI模型提取的技能使未见任务准确率提升30.0%。
原文摘要 · Abstract (English)
Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. Such models matter as computer-use agents enter real work, where agents need to learn how tasks are actually performed, and organizations need to audit and reuse that knowledge. However, inducing such task models is challenging, as activity is observed only as low-level events and real-world work is multi-threaded with interleaved goals. Existing methods assume a given task or a single workflow, and produce step-level summaries rather than structured task models. We introduce Task Model Induction (TMI), which (i) discovers the latent tasks in an unconstrained trace, disentangling concurrent activity, and (ii) for each latent task, induces a task model pairing a hierarchical objective model of recursive goal decomposition with a procedure model of the control flow that organized the execution. Intrinsically, on controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings and reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline. Extrinsically, skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。