通过过程评分发现大模型任务失败的真正原因
ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents
- 设计双维度评估框架，同时看结果和推理过程
- 4个维度评分揭示结果验证是主要瓶颈
- 适用于调试复杂长流程任务的AI代理
随着基于大语言模型的智能体被部署于复杂的多步骤工作流中，一个关键评估缺口浮现：现有基准仅评价最终结果，无法区分可靠推理与偶然成功，也无法定位长期任务中的具体失败环节。本文提出ClawTrack，一种双评估基准，同时衡量智能体达成的目标（任务得分）和实现路径（过程得分）。该基准包含8个领域的320个任务及25个以上确定性模拟服务。过程评分器对每个推理步骤从目标对齐、效率、信息利用和结果验证四个维度打分，依据12,541条特定任务评分标准。在超过16,000次实验中评估21个模型，发现：(1) 过程得分可有效将成败归因于具体推理维度，识别出仅靠结果评价无法察觉的侥幸通过；(2) 四个维度互补，其中结果验证是系统性瓶颈；(3) 框架对不同评估大模型的选择具有鲁棒性；(4) 基于过程的轨迹过滤在不同规模模型上均带来稳定训练改进。
原文摘要 · Abstract (English)
As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks. In this work, we present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves (Task Score) and how it achieves it (Process Score). ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services. A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification), anchored by 12,541 task-specific rubric items. Evaluating 21 models over 16,000+ trials, we find that: (1) process scores effectively attribute success and failure to specific reasoning dimensions, filtering lucky passes invisible to outcome-only evaluation; (2) the four dimensions are complementary, with result verification as the systematic bottleneck; (3) the framework is robust to evaluator choice across different judge LLMs; and (4) process-based trajectory filtering yields consistent post-training improvements across model scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。