无需标注失败步骤,仅用成功轨迹就能快速定位大模型智能体失败原因。
Tracing Agentic Failure from the Flow of Success

- 基于成功轨迹的动态模式学习,通过神经微分方程建模正常行为。
- 仅用100条成功轨迹训练,推理时对失败轨迹自动评分并定位错误步骤。
- 比提示法快200至5000倍,且在域内和域外数据上F1提升20%和7%。
大模型智能体系统的失败归因——即识别导致任务失败的具体步骤——对调试与改进至关重要。现有方法或依赖计算开销大的提示管道,或需在失败轨迹上进行带步骤级错误标注的后训练,后者成本高且难扩展。我们主张实用的失败归因模型应轻量且无需失败数据的步骤级监督。为此,提出无监督失败归因:仅在成功轨迹上训练,推理时判断失败轨迹中异常步骤。OAT将此问题转化为一类学习,利用神经控制微分方程建模成功轨迹在隐空间中的动态模式。推理时,每个步骤根据其偏离该动态模式的程度获得异常分数,进而识别错误步骤。实验表明,仅用100条成功轨迹训练,OAT比提示基基线快200至5000倍,在域内和域外数据集上分别取得+20%和+7%的F1提升,证明其是高效可靠的智能体故障诊断新方向。
原文摘要 · Abstract (English)
Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000 $\times$ faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。