构建时间一致的代码库评估基准,避免未来信息泄露。
ATime-Consistent Benchmark for Repository-Level Software Engineering Evaluation
- 在T0时刻快照代码库,仅用T0前的资料构建知识。
- 文件级F1最高达0.8081,提示粒度影响显著。
- 适合研究代码生成与智能编程工具的评估者。
现有代码库感知的软件工程系统评估常受合成任务设计、提示泄露和时间污染影响。本文提出一种时间一致的基准方法:在时间T0对代码库进行快照,仅使用T0前的产物构建代码知识,并在后续时间段(T0, T1]内合并的Pull Request衍生的任务上进行评估。每个历史PR通过LLM辅助的提示生成管道转化为自然语言任务,基准以匹配的A/B对照形式呈现,同一智能体在有无代码库知识条件下对比,其余变量保持不变。我们还报告了在DragonFly与React两个开源仓库上使用三个Claude系列模型及四种提示粒度的基线研究。在两个仓库中,文件级F1随提示从最小到引导逐步提升,最强模型分别达到0.8081(DragonFly)和0.8078(React)。结果表明提示构造是关键评估变量。更广泛地,该基准强调时间一致性与提示控制是代码库感知评估的核心有效性要求。
原文摘要 · Abstract (English)
Evaluation of repository-aware software engineering systems is often confounded by synthetic task design, prompt leakage, and temporal contamination between repository knowledge and future code changes. We present a time-consistent benchmark methodology that snapshots a repository at time T0, constructs repository-derived code knowledge using only artifacts available before T0, and evaluates on engineering tasks derived from pull requests merged in the future interval (T0, T1]. Each historical pull request is transformed into a natural-language task through an LLM-assisted prompt-generation pipeline, and the benchmark is formalized as a matched A/B comparison in which the same software engineering agent is evaluated with and without repository-derived code knowledge while all other variables are held constant. We also report a baseline characterization study on two open-source repositories, DragonFly and React, using three Claude-family models and four prompt granularities. Across both repositories, file-level F1 increases monotonically from minimal to guided prompts, reaching 0.8081 on DragonFly and 0.8078 on React for the strongest tested model. These results show that prompt construction is a first-order benchmark variable. More broadly, the benchmark highlights that temporal consistency and prompt control are core validity requirements for repository-aware software engineering evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。