arXiv:2608.09153cs.AIcs.LG2026-08

用历史交互记录自动诊断AI上下文错误,无需人工标注。

TRACE: TRajectory Attribution for Automated Context Engineering

  • 从历史轨迹中提取用户修正等隐式反馈信号,定位上下文故障点。
  • 在60条故障轨迹上实现72.7%根因定位准确率和82%修复成功率。
  • 适合需要自动化维护复杂AI系统的研发团队使用。

生产级AI代理在上下文源(系统提示、知识库、工具描述、流程技能)存在错误或缺失时会失效。当前维护依赖人工日志审查与临时调试,随交互量增长面临可扩展性瓶颈。本文提出TRACE(TRajectory Attribution for Automated Context Engineering),一个基于历史代理轨迹的自动化反馈闭环,用于诊断与修复上下文缺陷。核心洞察是轨迹中蕴含丰富的隐式不满信号——如用户修正、重述、放弃行为——可精准揭示上下文源的失败位置,无需显式反馈收集。不同于模型微调,TRACE作用于上下文层,支持快速迭代而无需重新训练。本文贡献四点:(1) 轨迹挖掘框架,系统化提取历史执行中的诊断信息;(2) 多组件因果归因,将文本梯度从单一提示优化拓展至异构上下文源(技能、知识库、工具、提示);(3) 探索性验证机制,让代理主动读取上下文以区分需创建的新内容与需更新的过时内容,实现96%操作准确率;(4) 可复用的仿真方法与可验证基准,填补上下文调试缺乏公开数据集的空白,包含六类故障分类、真实标签与跨层验证协议。在覆盖三个复杂度层级(最多16个执行节点)、共60条不满轨迹的测试中,TRACE实现72.7%根因归因率与82%端到端修复有效性,表明超过80%的上下文层故障可通过挖掘历史轨迹自动诊断与修复,这一资源在生产系统中长期被忽视。

原文摘要 · Abstract (English)

Production AI agents fail when their context sources -- system prompts, knowledge bases, tool descriptions, and procedural skills -- contain errors or gaps. Current maintenance relies on manual log review and ad-hoc debugging, creating a scalability bottleneck as interaction volume grows. We present TRACE (TRajectory Attribution for Automated Context Engineering), an automated feedback loop that mines historical agent trajectories to diagnose and remediate context failures. Our key insight is that trajectories are rich with implicit dissatisfaction signals -- user corrections, rephrasing, abandonment cues -- that reveal precisely where context sources failed, without explicit feedback collection. Unlike model fine-tuning, TRACE operates on the context layer, enabling rapid iteration without retraining. We make four contributions: (1) a trajectory mining framework that systematically extracts diagnostic information from historical agent executions; (2) multi-component causal attribution that extends textual gradients from monolithic prompt optimization to heterogeneous context sources (skills, knowledge bases, tools, prompts); (3) exploratory verification, where agents actively read context sources to distinguish content gaps requiring CREATE from stale content requiring UPDATE, achieving 96% operation accuracy; and (4) a reusable simulation methodology and verifiable benchmark addressing the absence of open datasets for context debugging, with a six-category fault taxonomy, ground truth annotations, and a cross-layer verification protocol. On 60 dissatisfaction traces spanning three complexity tiers (up to 16 execution nodes), TRACE achieves 72.7% root cause attribution and 82% end-to-end fix effectiveness, showing that over 80% of context-layer failures can be automatically diagnosed and remediated by mining historical trajectories, an overlooked resource in production systems.

上下文工程自动化诊断轨迹分析AI运维

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。