arXiv:2605.20086cs.NEcs.AI2026-05被引 3

揭示进化编程代理实际演化的机制,发现多数进步来自重复代码重用而非新结构。

What Do Evolutionary Coding Agents Evolve?

论文配图:What Do Evolutionary Coding Agents Evolve?
图 1 · 摘自论文原文
  • 通过追踪演化过程,识别代码修改的九类常见模式。
  • 30%新增代码是之前删除代码的字节级重复,表明存在确定性循环。
  • 提出EvoReplay方法,可控制干预验证不同演化机制的影响。

近期研究将大语言模型与进化搜索结合,通过任务反馈迭代生成、修改和选择代码,在数学发现与算法设计中取得优异成果。然而核心问题仍存:这些系统究竟演化了什么?现有评估仅关注最终得分,但得分可能源于新算法结构、策略微调、内部知识重组或评价器过拟合。区分这些机制需分析搜索过程本身。本文构建EvoTrace数据集,涵盖四种演化框架、推理与非推理模型及16个数学与算法任务的演化轨迹。提出EvoReplay方法,通过回放重建高分解的局部状态,并实施常数调整、组件移除、模型或提示替换等可控干预。利用大模型为裁判的管道标注每条代码修改类型,经盲评验证。结果显示,绝大多数得分提升来自少数几类编辑;约30%新增代码为此前删除代码的字节级重引,贯穿几乎每次运行。这表明基准性能提升可能源于不同机制,其中仅部分对应真正的新算法结构。EvoTrace使演化编程代理的评估超越最终分数,实现更诊断性分析。

原文摘要 · Abstract (English)

Recent work pairs LLMs with evolutionary search to iteratively generate, modify, and select code using task-specific feedback. These systems have produced strong results in mathematical discovery and algorithm design, yet a fundamental question remains: what do they actually evolve? Progress is typically summarized by the best score a run reaches under a task-specific evaluator, but that score can reflect several different mechanisms: new algorithmic structure, re-tuning an existing strategy, recombining ideas already in the model's internal knowledge, or overfitting to the evaluator. Distinguishing these mechanisms requires inspecting the search process itself, not only its final outcome. We introduce EvoTrace, a dataset of evolutionary coding traces spanning four evolutionary frameworks, reasoning and non-reasoning models, and 16 tasks across mathematics and algorithm design. To analyze these traces, we develop EvoReplay, a replay-based methodology that reconstructs the local search states behind high-scoring solutions and tests controlled interventions, including adjusting constants, removing program components and substituting models or prompting contexts. We annotate every code edit in EvoTrace with one of nine recurring edit types using an LLM-as-judge pipeline validated against blind human re-annotation. Across EvoTrace, most score gains come from a small subset of these edit types. We further find a deterministic cycling pattern: about 30% of code lines added during search are byte-identical re-introductions of previously-deleted lines, present throughout nearly every run. These results show that benchmark gains in evolutionary coding agents can arise from qualitatively different mechanisms, only some of which correspond to new algorithmic structure. EvoTrace enables more diagnostic evaluation of evolutionary coding agents beyond final benchmark scores.

进化计算代码生成机制分析LLM应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。