用进化方法自动生成更难的编程题目,突破现有评测瓶颈。
BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

- 基于参考解法结构化演化,生成语义可执行的新题目
- 新题难度显著提升,顶尖模型通过率降至27.5%~62.6%
- 适合需要高难度评测与训练信号的研究者使用
前沿大语言模型的快速进步导致现有评测数据集普遍饱和,难以区分模型能力或提供有效训练信号。例如,在LiveCodeBench上,前沿模型在简单任务上的通过率超过99%,平均通过率超过90%。构建新难题通常需大量人工投入,成为进展瓶颈。我们提出BenchEvolver,一种以解法为中心的进化框架,可自动将现有编程题转化为更难变体。不从零生成题目,而是对参考解法进行结构化演化,并从中推导出对应题干与测试用例。该设计基于可执行语义,实现高质量、多样化且困难的任务规模化构建。在LiveCodeBench和SciCode上应用后,生成的任务显著更难,同时保持有效性、参考正确性与多样性。我们进一步构建了包含91个题目的LiveCodeBench-Plus基准,融合演化题与原版LCB-v6难题,其中前沿模型通过率在27.5%至62.6%之间,重新恢复强模型间的区分度。重要的是,生成模型自身仍无法轻松解决所生成任务,支持自我改进。进一步实验显示,在演化后的LCB任务上进行强化学习,可带来显著性能提升:gpt-oss-20b在LCB v6 Hard和LCB-Pro Easy上分别获得+8.7和+8.3的通过率提升,优于仅使用原始种子数据的提升幅度70.7%和34.8%。结果表明,BenchEvolver能将饱和评测集转化为前沿级评估体系与可复用的训练信号。
原文摘要 · Abstract (English)
The rapid progress of frontier large language models has led to widespread benchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide useful training signal. For instance, on LiveCodeBench, frontier models achieve over 99% Pass@1 on easy splits and exceed 90% Pass@1 on average across difficulty levels. Constructing new, challenging datasets typically requires substantial human effort, creating a bottleneck for progress. We introduce BenchEvolver, a solution-centric evolutionary framework that automatically transforms existing coding problems into harder variants. Rather than generating problems from scratch, BenchEvolver evolves reference solutions through structured transformations and derives corresponding statements and tests from the evolved solutions. This design grounds generation in executable semantics, enabling scalable construction of high-quality, diverse, and difficult tasks with verifiable correctness. Applying BenchEvolver to LiveCodeBench and SciCode, we obtain evolved tasks that are substantially harder while maintaining validity, reference correctness, and diversity. We further curate LiveCodeBench-Plus, a 91-problem benchmark combining evolved and difficult original LCB-v6 tasks, where frontier-model Pass@1 ranges from 27.5% to 62.6%, restoring clear discrimination among strong coding models. Importantly, evolved tasks remain challenging even for the model that generates them, enabling self-improvement. We further show that RL on evolved LCB tasks improves held-out coding performance: for gpt-oss-20b, seed+evolved training achieves +8.7 and +8.3 Pass@1 gains on LCB v6 Hard and LCB-Pro Easy, exceeding seed-only gains by 70.7% and 34.8%, respectively. Our results show that BenchEvolver can convert saturated benchmarks into frontier-level evaluation suites and reusable training signal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。