用图结构分析智能体轨迹,揭示隐藏的决策差异与失败模式。
TraceGraph: Shared Decision Landscapes for Diagnosing and Improving Agent Trajectories

- 构建共享决策图谱,将多模型轨迹转化为可观测状态的图结构。
- 发现不同任务评估侧重避免陷阱或从陷阱中恢复,且修复策略提升成功率至44.8%。
- 适合研究智能体行为差异、失败原因及改进策略的开发者和评测人员。
智能体基准测试越来越多地记录丰富的交互轨迹,但评估通常仅简化为通过率或奖励分数。我们提出TraceGraph,一种基于图的框架,将公开的多模型智能体轨迹转化为共享决策景观。对于每个任务,TraceGraph在引入模型身份前,对合并的轨迹中可观测的动作-观测状态构建图结构,并叠加以结果为导向的高效核心与陷阱区域。每条轨迹被总结为三个事件:访问、陷入陷阱和修复。在五个基准分割的轨迹中,TraceGraph分析揭示了聚合评分掩盖的导航差异,表明不同分割任务更倾向奖励避免陷阱或从陷阱中恢复。该图谱还启发了一种针对SWE-bench的陷阱感知修复流程:运行时检测器在匹配历史陷阱区域的状态触发,随后评估基于相同前缀的轻量级延续策略。在触发状态上,最优的单因素策略使各提供商子集的官方解决率从40.4%提升至43.5%,共触发实例从41.0%提升至44.8%,并结合了提供商特定的活跃组件。总体而言,TraceGraph为探讨智能体基准测试的本质、模型在共享景观上的分歧以及如何利用失败区域指导后续改进提供了过程语言。
原文摘要 · Abstract (English)
Agent benchmarks increasingly record rich interaction trajectories, yet evaluation often reduces each rollout to a pass rate or reward score. We introduce TraceGraph, a graph-based framework that turns released multi-model agent trajectories into shared decision landscapes. For each task, TraceGraph builds a graph over observable action-observation states from pooled rollouts before model identity is introduced. It then overlays outcome-informed productive cores and trap regions, and summarizes each rollout with three events: Access, Trap exposure, and Repair. Across trajectories spanning five benchmark splits, TraceGraph profiles reveal navigation differences hidden by aggregate scores and show that splits differ in whether they reward avoiding traps or recovering from them. The same TraceGraph landscape also motivates a trap-aware recovery pipeline for SWE-bench: aruntime detector fires on states matching historical trap regions, then lightweight continuation policies are evaluated from the same prefix. On fired states, the best pooled single-factor policy raises official resolved rate from 40.4% to 43.5% on the per-provider fired subset and from 41.0% to 44.8% on common-fired instances, with provider-specific active components. Overall, TraceGraph provides a process vocabulary for asking what agent benchmarks test, where models diverge on a shared landscape, and how failure regions can guide downstream improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。