用合成编辑序列训练模型，让代码生成更贴近工程师实际编辑习惯。
Training Language Models on Synthetic Edit Sequences Improves Code Synthesis
- 通过语法检查器生成程序的合成编辑序列，模拟真实编码过程。
- 在HumanEval等数据集上，新模型在pass@1和更高pass@k指标上表现更优。
- 适合研究代码生成、编程辅助工具或想提升模型可解释性的开发者。
软件工程师主要通过修改已有代码来编写程序，而语言模型（LMs）则采用单次自回归方式生成代码。这一差异部分源于序列化编辑数据的稀缺性。尽管高质量的代码合成指令数据已较难获取，用于合成的编辑数据更是极为稀少。为此，我们提出一种名为LintSeq的合成数据生成算法，利用代码检查器对源码中相互依赖的代码行进行过程式采样，将程序重构为合成编辑序列。这些合成编辑序列能反映编程语言的语法与语义。我们使用该算法将指令+程序对数据集重构为指令+程序编辑序列对，并在此基础上微调多个规模从2.6B到14B参数的语言模型。在HumanEval、MBPP(+)、CodeContests、DS-1000和BigCodeBench上进行全面评估，结果显示，基于编辑序列训练的模型在pass@1上达到或超越基线，且随着测试时总浮点运算量增加，其pass@k表现呈现更好的扩展性。此外，我们还预训练了小型代码理解模型，将其微调为逐编辑生成代码后，在HumanEval和MBPP(+)上表现优于同规模的CodeT5+、AlphaCode和Codex等现有模型。
原文摘要 · Abstract (English)
Software engineers mainly write code by editing existing programs. In contrast, language models (LMs) autoregressively synthesize programs in a single pass. One explanation for this is the scarcity of sequential edit data. While high-quality instruction data for code synthesis is scarce, edit data for synthesis is even scarcer. To fill this gap, we develop a synthetic data generation algorithm called LintSeq. This algorithm refactors programs into sequences of synthetic edits by using a linter to procedurally sample across interdependent lines of source code. Synthetic edits sampled with LintSeq reflect the syntax and semantics of their programming language. To test the algorithm, we use it to refactor a dataset of instruction + program pairs into instruction + program-diff-sequence tuples. Then, we fine-tune a series of smaller LMs ranging from 2.6B to 14B parameters on both the re-factored and original versions of this dataset. We perform comprehensive evaluations comparing edit sequence code LMs against baselines on HumanEval, MBPP(+), CodeContests, DS-1000, and BigCodeBench. We show that models fine-tuned to iteratively synthesize code match or outperform baselines on pass@1, and exhibit better scaling across higher pass@k as a function of total test-time FLOPs. Finally, we also pretrain our own tiny LMs for code understanding. We show that fine-tuning these models to synthesize code edit-by-edit results in strong performance on HumanEval and MBPP(+) compared to existing code language models of similar scale such as CodeT5+, AlphaCode, and Codex.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。