新评估框架可全面衡量智能体推理过程的效率与质量。
TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents
- 基于轨迹的分层效用函数，量化推理过程的效率与认知质量。
- 通过最小引导需求测试，揭示智能体潜在能力与鲁棒性。
- 在可控复杂度数据集上验证，发现单一指标无法捕捉关键权衡。
深度研究智能体的评估面临重大挑战，传统以结果为导向的指标无法捕捉其复杂推理的细微差别。当前评估存在两大问题：一是过度依赖单一指标（如Pass@1），导致“高分幻觉”，忽视推理过程的质量、效率和合理性；二是静态基准无法量化鲁棒性和潜在能力等关键属性。为此，本文提出TRACE（轨迹感知综合评估）框架，全面评估问题求解全过程。为克服“高分幻觉”，引入分层轨迹效用函数，量化过程效率与认知质量（包括证据锚定）及准确性。为测量深层属性，提出分级能力评估协议，通过确定成功所需的最小引导量来衡量智能体的潜在能力。贡献包括TRACE框架、新指标及配套的DeepResearch-Bench数据集（具备可控复杂度）。实验表明，TRACE能生成精细排名，揭示智能体在准确率、效率与鲁棒性之间的关键权衡，而这些是单一指标完全忽略的。
原文摘要 · Abstract (English)
The evaluation of Deep Research Agents is a critical challenge, as conventional outcome-based metrics fail to capture the nuances of their complex reasoning. Current evaluation faces two primary challenges: 1) a reliance on singular metrics like Pass@1, creating a "high-score illusion" that ignores the quality, efficiency, and soundness of the reasoning process; and 2) the failure of static benchmarks to quantify crucial attributes like robustness and latent capability. To address these gaps, we introduce TRACE (Trajectory-Aware Comprehensive Evaluation), a framework that holistically assesses the entire problem-solving trajectory. To counter the "high-score illusion", we propose a Hierarchical Trajectory Utility Function that quantifies process efficiency and cognitive quality, including evidence grounding, alongside accuracy. To measure deeper attributes, TRACE introduces a Scaffolded Capability Assessment protocol, quantifying an agent's latent ability by determining the minimum guidance needed for success. Our contributions include the TRACE framework, its novel metrics, and the accompanying DeepResearch-Bench with controllable complexity. Experiments show TRACE delivers a granular ranking that uncovers critical trade-offs between agent accuracy, efficiency, and robustness entirely missed by singular metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。