用有效反馈计算衡量智能体表现，比单纯看消耗更准。
Scaling Laws for Agent Harnesses via Effective Feedback Compute
- 提出有效反馈计算（EFC）衡量反馈质量，排除冗余与无效交互。
- 真实场景中EFC相关性达R²=0.93，远超原始计算指标。
- 提升智能体任务通过率至68.2%，成本降为原来的四成，适合优化智能体系统。
智能体约束框架通过控制工具调用、反馈、验证、记忆和修复来影响语言模型性能。然而，仅以测试时的资源消耗（如令牌数、工具调用次数、运行时间或成本）无法区分有用反馈与冗余或不稳定的交互。本文提出「有效反馈计算」（Effective Feedback Compute, EFC），作为衡量信息丰富、有效、非冗余且被保留的反馈的细粒度尺度。进一步定义了估算型EFC（Estimated-EFC）、非冗余稳定型EFC（NRS-EFC）、约束效率η及任务需求归一化，适用于真实轨迹与异构任务。在合成、真实、预留及未来评估中，基于EFC的尺度显著优于原始计算基线和SAS方法。在受控缩放实验中，理想EFC/$D_{\mathrm{task}}$达到R²=0.99；在真实轨迹上，NRS-EFC/$D_{\mathrm{task}}$实现R²=0.93，而原始计算相关性接近零或为负。最后，本方法作为现有约束框架的辅助控制层，使平均通过率从61.2%提升至68.2%，同时将平均原始成本从213.8降至85.1，且在相同设定下实现。结果表明，约束系统的可扩展性取决于持久、任务充分的反馈，而非单纯的计算投入。
原文摘要 · Abstract (English)
Agent harnesses shape language-model performance by controlling tool use, feedback, verification, memory, and repair. Yet raw test-time expenditure, such as tokens, tool calls, wall time, or cost, cannot distinguish useful feedback from redundant or unstable interaction. We introduce \emph{Effective Feedback Compute} (EFC), a trace-level scaling coordinate for informative, valid, non-redundant, and retained feedback. We further define Estimated-EFC, NRS-EFC, harness efficiency $η$, and task-demand normalization for realistic traces and heterogeneous tasks. Across synthetic, real, held-out, and prospective evaluations, EFC-based coordinates outperform raw-compute baselines and SAS. Oracle-EFC/$D_{\mathrm{task}}$ reaches $R^2=0.99$ in controlled scaling, and NRS-EFC/$D_{\mathrm{task}}$ reaches $R^2=0.93$ on real traces where raw compute has near-zero or negative fit. Finally, \ours uses EFC as a companion control layer for existing harnesses, improving mean pass rate from $61.2\%$ to $68.2\%$ while reducing mean raw cost from $213.8$ to $85.1$ under matched settings. These results suggest that harness scaling depends on durable, task-sufficient feedback rather than raw computation alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。