让编程助手学会记用户改正,自动执行避免重复犯错。
Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents

- 从用户对话修正中提取规则,动态生成运行时检查
- 在分布内任务上违规率从100%降至37.6%,分布外降至2.0%
- 适合需要长期记忆用户偏好、减少重复纠错的智能编程助手
交互式大模型代理正融入日常开发,但其表现不会随使用时间改善:某次会话中记住的修正,下次仍可能被违反。我们研究了偏好感知与偏好遵守之间的差距。基于匿名化真实用户摩擦案例的任务显示,Mem0记忆系统仍有57.5%的适用偏好检查未被遵守。本文提出测试时规则获取与编译执行(TRACE),一种可直接部署于编程代理运行时的技能层流水线,能挖掘用户修正,将其重写为原子规则,并编译成运行时检查,确保未来任务前通过。与开发者预先编写检查不同,TRACE的规则源自用户自身对话中的修正。我们在ClawArena编码任务及MemoryArena衍生的记忆密集型任务上进行模拟用户在环实验。在ClawArena上,分布内任务违规率从100.0%降至37.6%,分布外任务降至2.0%;在记忆密集任务上,分布内违规率从100.0%降至60.5%,同时任务通过率匹配或超越最强记忆基线。结果表明,将用户修正编译为运行时强制执行,可有效解决仅靠记忆无法可靠缓解的重复摩擦问题,减少用户跨会话重复说明相同修正的需求。实验代码见https://github.com/YujunZhou/TRACE_exp,可部署技能见https://github.com/YujunZhou/tellonce。
原文摘要 · Abstract (English)
Interactive LLM agents are becoming part of daily work, but they do not reliably become easier to work with over time: a correction remembered in one session may still be violated in the next. We study this gap between preference access and preference compliance. In tasks derived from anonymized real-user friction cases, Mem0 memory still leaves 57.5% of applicable preference checks violated. We introduce Test-time Rule Acquisition and Compiled Enforcement (TRACE), a drop-in skill-layer pipeline for coding-agent runtimes that mines user corrections, rewrites them as atomic rules, and compiles them into runtime checks that must pass before an agent completes future tasks. Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections. We evaluate TRACE with simulated user-in-the-loop experiments on ClawArena coding-agent tasks and MemoryArena-derived memory-intensive tasks. On ClawArena, TRACE reduces held-out preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks. On MemoryArena-derived tasks, TRACE reduces in-distribution violation from 100.0% to 60.5% while matching or exceeding the strongest memory baseline on task pass. These results suggest that compiling corrections into runtime enforcement can address a repeated-friction failure mode that memory alone does not reliably solve, reducing the need for users to restate the same correction across future sessions. Experiment code is available at https://github.com/YujunZhou/TRACE_exp, and the deployable skill is available at https://github.com/YujunZhou/tellonce.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。