arXiv:2602.10171cs.SEcs.AI2026-02被引 1

评测大模型编程系统在迭代中自我进化的能力,对比人类表现。

EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems

  • 设计动态评估框架,追踪模型解题过程中的正确率与效率变化。
  • 发现自进化系统随迭代提升效率,解决时间与内存消耗持续下降。
  • 支持多语言对比,可分析长尾语言稳定性与人类表现基准。

随着大语言模型在编程任务中的进步,基于LLM的编程系统已从一次性生成演变为具备推理时迭代优化能力的复杂系统。然而,现有代码评测基准主要关注静态正确性,隐含假设模型能力在推理过程中固定不变,无法捕捉准确性与效率随迭代改进的现象。同时,这些基准对资源开销缺乏记录,且很少将模型性能与人类程序员进行直接比较。此外,多数基准集中于高资源语言,忽视跨语言鲁棒性与长尾语言稳定性。为此,我们提出EvoCodeBench,一个面向跨语言自进化LLM驱动编程系统的基准,支持与人类程序员的直接性能对比。该基准追踪解题过程中的性能动态,测量正确率及求解时间、内存消耗、算法设计改进等效率指标。通过人类中心化参照体系,实现对模型表现相对于人类能力分布的相对评估。同时,支持多语言统一协议,促进系统性跨语言与长尾语言稳定性分析。实验表明,自进化系统在时间与资源效率上均呈现可观测提升,而人类相对分析与多语言视角揭示了仅看准确率无法获得的深层洞察。EvoCodeBench为评估演化型编程智能奠定了基础。

原文摘要 · Abstract (English)

As large language models (LLMs) continue to advance in programming tasks, LLM-driven coding systems have evolved from one-shot code generation into complex systems capable of iterative improvement during inference. However, existing code benchmarks primarily emphasize static correctness and implicitly assume fixed model capability during inference. As a result, they do not capture inference-time self-evolution, such as whether accuracy and efficiency improve as an agent iteratively refines its solutions. They also provide limited accounting of resource costs and rarely calibrate model performance against that of human programmers. Moreover, many benchmarks are dominated by high-resource languages, leaving cross-language robustness and long-tail language stability underexplored. Therefore, we present EvoCodeBench, a benchmark for evaluating self-evolving LLM-driven coding systems across programming languages with direct comparison to human performance. EvoCodeBench tracks performance dynamics, measuring solution correctness alongside efficiency metrics such as solving time, memory consumption, and improvement algorithmic design over repeated problem-solving attempts. To ground evaluation in a human-centered reference frame, we directly compare model performance with that of human programmers on the same tasks, enabling relative performance assessment within the human ability distribution. Furthermore, EvoCodeBench supports multiple programming languages, enabling systematic cross-language and long-tail stability analyses under a unified protocol. Our results demonstrate that self-evolving systems exhibit measurable gains in efficiency over time, and that human-relative and multi-language analyses provide insights unavailable through accuracy alone. EvoCodeBench establishes a foundation for evaluating coding intelligence in evolving LLM-driven systems.

编程智能自进化多语言人机对比

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。