将用户操作记录转为可复现的结构化记忆,让智能体高效回溯任务。
Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay
- 用无模型流水线将屏幕活动切分为带类型和证据指针的事件帧。
- 压缩后数据量减少86倍,问答准确率达98.4%,优于大模型摘要。
- 首次量化任务开销与重复率,适合需要可审计代理的场景。
计算机使用代理因无法直接获取用户行为而重复推导已执行流程。本文提出一种无模型、确定性的流水线,将被动捕获的屏幕活动编译为带有应用、站点、时间、输入量及原始数据指针的结构化活动帧,输出字节一致、可缓存且可机械验证。在一位用户51天内共128,756帧的数据上,该编译器将一天原始捕获压缩至86倍小的提示上下文块,耗时仅68毫秒;基于该块的代理问答准确率达98.4%(威尔逊95%置信区间91.7%-99.7%),显著优于同等捕获的大语言模型摘要(66-80%)。同一编译器还可作为成本分析工具,测得任务开销比R为60-343倍,任务重复率在样本内为9.0%,样本外为7.7%,全舰队令牌上限可控制在8%左右。编译后的任务可零模型令牌实现确定性重演,已在真实场景中演示成功。方案代码与评估工具均已开源。
原文摘要 · Abstract (English)
Computer-use agents pay full frontier inference to re-derive routines their user has already performed, because an agent's memory today records what the user said, not what the user did. We compile passively captured screen activity into agent memory with a deterministic, zero-model pipeline: it segments a local capture stream into typed activity frames, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows, with no model in the loop, so the output is byte-identical, cacheable, and mechanically auditable. On one professional's single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms, and an agent reading that block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%) against an independent oracle, versus 66-80% for an LLM summary of the same capture, a mid-tier model reading the block matching a frontier one. The same compiler doubles as a demand-side cost instrument. Read off passive, pre-delegation human activity rather than agent rollouts, it supplies two parameters that agent-cost models assume but, to our knowledge, have not measured: the Routine Overhead Ratio R and the routine recurrence h. We report first values of R, a modeled upper bound, at 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, for a realistic all-fleet token ceiling near 8%; a compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit. Schema, compiler, and evaluation harness are open.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。