用沙盒测试构建可扩展的代码仓库级执行环境，提升生成效果。
RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing
- 通过沙盒隔离目标函数及其依赖，降低环境构建复杂度。
- 在HumanEval上提升5.5% Pass@1，RepoEval上提升3.5% Pass@1。
- 适合需要真实执行反馈的代码生成研究与模型评估者。
我们提出RepoST，一种可扩展的方法，用于构建为代码仓库级生成提供执行反馈的环境，适用于训练和评估。与现有工作需完整构建仓库以实现执行不同，RepoST采用沙盒测试机制，将目标函数及其依赖隔离至独立脚本中进行测试，显著降低外部依赖复杂度，支持大规模环境构建。我们基于该方法构建了包含832个仓库中7,415个函数的训练数据集RepoST-Train。使用该数据集训练后，模型在HumanEval上的Pass@1提升5.5%，在RepoEval上提升3.5%。同时，我们还构建了评估数据集RepoST-Eval，对12个代码生成模型进行了基准测试。
原文摘要 · Abstract (English)
We present RepoST, a scalable method to construct environments that provide execution feedback for repository-level code generation for both training and evaluation. Unlike existing works that aim to build entire repositories for execution, which is challenging for both human and LLMs, we provide execution feedback with sandbox testing, which isolates a given target function and its dependencies to a separate script for testing. Sandbox testing reduces the complexity of external dependencies and enables constructing environments at a large scale. We use our method to construct RepoST-Train, a large-scale train set with 7,415 functions from 832 repositories. Training with the execution feedback provided by RepoST-Train leads to a performance gain of 5.5% Pass@1 on HumanEval and 3.5% Pass@1 on RepoEval. We also build an evaluation dataset, RepoST-Eval, and benchmark 12 code generation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。