通过提前预测任务结果，大幅降低大模型智能体评估成本。
EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction
- 在任务执行中途基于行为特征提前判断成败，提前终止耗时步骤。
- 在三个基准上减少13%-26%的执行步数，最多节省44.1%输入token。
- 适合需要频繁迭代评估的大模型开发团队，尤其关注成本控制者。
评估大语言模型智能体对指导其发展至关重要，但成本已变得极高：单次前沿模型在智能体基准上的运行成本可达数百至数千美元，且需在迭代开发中重复进行。此前工作聚焦于基准压缩，虽减少了评估任务数量，但未降低每个任务的执行开销。本文提出早期结果预测（EarlyOutcomePrediction），从任务内部效率切入，实现成本削减。核心洞察是：智能体的最终结果往往在其执行过程中期便已可判别。我们构建了EarlyEval框架，使用LightGBM训练成功与失败两个轻量分类器，基于行为、文本和参考解特征，在任一分类器达到校准置信阈值时立即终止任务，每步开销极低。在SWE-bench Verified、TerminalBench和Toolathlon三个基准上，EarlyEval可减少13%-26%的执行步数，最多节省44.1%输入token与29.4%输出token，预测准确率达89%-97%，对平均每个智能体的解决率影响仅±1-2个百分点。
原文摘要 · Abstract (English)
Evaluating LLM agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars, a price paid repeatedly across iterative development cycles. Prior efforts, centered on benchmark distillation, reduce the number of evaluation tasks but leave the cost of executing each retained task untouched. In this work, we introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task. Our key insight is that an agent's final outcome is often evident from its intermediate behavior well before execution completes. We instantiate this idea in EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features, and halts an agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. Across three benchmarks, SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval can eliminate 13%-26% of agent steps and up to 44.1% input tokens and 29.4% output tokens at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。