不执行模拟就能预判任务难易度，提升评估与训练效率
Predicting Task Difficulty Without Rollouts
- 从任务描述直接预测难度，无需试错模拟
- 发现词元熵是有效的预测信号，AUC指标可能掩盖问题
- 可检测环境缺陷，适合评测设计与课程构建者
任务难度决定智能体成功概率。在状态化环境中，无需执行模拟即可从任务描述中直接预测难度，将极大提升效率。可靠估计有助于环境设计者校准评估基准，并构建渐进式训练课程。随着智能体进入长时序领域，传统试错法成为严重计算瓶颈。现有早期预测研究多局限于静态任务或孤立编程环境，依赖狭窄特征且评价指标不准。本文在17个智能体基准上研究事前难度预测，涵盖编码、数学、机器学习、网页导航、函数调用等。结果表明，AUC可能掩盖差的预测表现；词元级熵是有效预测信号；预期与实际难度残差可揭示隐藏环境问题，如数据污染和不可行性。
原文摘要 · Abstract (English)
Task difficulty dictates an agent's likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments. Reliable estimates would therefore allow environment designers to calibrate evaluation benchmarks and construct progressive training curricula. This becomes increasingly important as agents move into long-horizon domains, where empirical trial-and-error is a severe computational bottleneck. Prior work on early prediction is limited to static tasks or isolated coding environments, often relying on narrow features and inaccurate evaluation metrics. We study \textit{ex ante} difficulty prediction across 17 agentic benchmarks spanning coding, mathematics, machine learning, web navigation, function calling, and other domains. We show that AUC can mask poor difficulty estimates, identify token-level entropy as a useful predictive signal, and show how residuals between expected and observed difficulty can expose hidden environment flaws such as contamination and infeasibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。