arXiv:2607.16387cs.SEcs.AI2026-07

用自动生成的失败分类体系,让智能体系统更高效地自我改进。

Fantastic Adaptive Taxonomies and How to Use Them

论文配图:Fantastic Adaptive Taxonomies and How to Use Them
图 1 · 摘自论文原文
  • 从执行轨迹自动构建三轴失败分类体系,无须人工标注。
  • 在多个任务上提升诊断准确率,最高使解决率提升10.7个百分点。
  • 适合需要持续优化的智能体系统,尤其适用于自动化调试与评估。

智能体系统的执行轨迹记录了其失败过程,而无需修改模型权重的优化方法(如轨迹选择、提示与流程优化、运行时监控)依赖这些轨迹获取反馈。然而原始轨迹信息冗长、依赖实例且缺乏稳定失败词汇表。本文提出AdaMAST,通过将目标系统的轨迹转化为紧凑、证据支持的失败分类体系:以系统级、角色特定和领域特定为固定维度,命名失败代码,并基于轨迹自动推导每个代码的定义与证据模式,全程无需人工编写或标注。该分类体系不仅是事后诊断工具,更是共享反馈接口,在三项任务中均表现优异:在代理系统搜索中,使用分类代码诊断失败候选者的表现优于自由文本反思;运行时反馈使SWE-agent在SWE-bench Verified Mini上的解决率从60%提升至70%,并使Claude Code的运行时技能从64.0%提升至70.7%;在轨迹选择中,基于分类代码的验证器AdaMAST-Judge在Terminal-Bench 2.0上比Pass@1提升8-15分。该词汇表压缩一个数量级,保留轨迹差异,且与专家标注高度一致,具备自适应性(不同领域间共享代码极少)。自适应失败分类体系实现了智能体输出与其改进机制之间的闭环。

原文摘要 · Abstract (English)

An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback. Yet raw traces are a poor medium for accumulating that feedback: long, instance-specific, and lacking a stable vocabulary for recurring failures. We argue that an agent system should instead maintain an explicit representation of how it fails, induced from its own behavior and reusable wherever failure feedback is needed. AdaMAST builds this representation by converting a target system's traces into a compact, evidence-grounded failure taxonomy: named failure codes organized along three fixed axes (system-level, role-specific, and domain-specific), with every name, definition, and evidence pattern induced from the traces; no code is hand-authored, no trace human-annotated. The taxonomy is not merely a post-hoc diagnostic but a shared feedback interface, improving agents in three ways. In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form reflection on all five benchmarks we test. At runtime, taxonomy feedback raises SWE-agent's resolution on SWE-bench Verified Mini from 60% with free-text reflection to 70%, and improves Claude Code from 64.0% to 70.7% as a runtime skill. In trajectory selection, AdaMAST-Judge, a verifier built on the induced codes, improves best-of-5 accuracy on Terminal-Bench 2.0 by 8-15 points over Pass@1. The vocabulary itself is compact (an order-of-magnitude compression that preserves trace distinctions), human-faithful (matching expert failure annotations more closely than a hand-crafted reference vocabulary), and adaptive (taxonomies induced for different domains share few codes). Adaptive failure taxonomies close the loop between the traces agents produce and the procedures that improve them.

智能体系统失败诊断自适应分类自动化优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。