arXiv:2608.20169cs.CLcs.AI2026-08

通过动态选择关键测试任务,让大模型测试框架优化快80%且不损失效果。

Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection

论文配图:Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
图 1 · 摘自论文原文
  • 根据候选框架的分歧程度,自适应挑选最具区分力的测试任务。
  • 在只评估1/5任务的情况下,仍能准确预测全集性能,最终效果媲美全量评估。
  • 适合追求高效测试框架优化的研究者和工程团队。

我们提出一种新型高效大模型测试框架优化方法——Task-CoEvolve,通过自适应选择验证任务实现。传统方法每轮均完整评估固定验证集,成本高昂,尤其对随框架演进而失去判别力的任务浪费严重。Task-CoEvolve通过两个核心机制解决:一是基于历史表现差异,用方差加权采样聚焦位于能力边界附近的任务;二是通过采样概率校正,从部分评估结果推断全集性能,确保各轮间可比性。在在线文本分类与Terminal-Bench 2.1上的实验表明,该方法持续优于子集基线,最终性能达到全集搜索水平,同时将评估次数减少80%。代码将开源。

原文摘要 · Abstract (English)

We present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose $\textbf{Task-CoEvolve}$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms subset-based baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.

框架优化自适应采样效率提升

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。