arXiv:2503.03656cs.SEcs.LG2025-03被引 2

让代码模型学会多样编辑,还能保持原有生成能力。

Robust Learning of Diverse Code Edits

  • 用合成数据生成+梯度筛选权重,实现高效适配
  • 新模型在5个基准上超越同尺寸及更大模型
  • 保留原始代码生成能力,适合工程落地使用

软件工程中频繁涉及代码修改,但现有代码语言模型难以应对多样化的编辑需求。本文提出一种新型合成数据生成管道和鲁棒模型适配算法SeleKT,基于种子代码与多样化编辑要求生成高质量样本(含原始/修改代码及自然语言指令)。为避免微调损失原始能力,SeleKT通过密集梯度定位关键权重,并稀疏投影回基模型以防止过拟合。基于此方法,我们推出下一代模型NextCoder(由QwenCoder-2.5适配而来),在五个代码编辑基准上表现优异,优于同规模及部分更大模型。该方法在DeepSeekCoder与QwenCoder两个模型家族中均具通用性,且验证了微调后仍保有代码生成与通用问题求解能力。模型、数据集与代码已开源:https://aka.ms/nextcoder。

原文摘要 · Abstract (English)

Software engineering activities frequently involve edits to existing code. However, contemporary code language models (LMs) lack the ability to handle diverse types of code-edit requirements. In this work, we attempt to overcome this shortcoming through (1) a novel synthetic data generation pipeline and (2) a robust model adaptation algorithm. Starting with seed code examples and diverse editing criteria, our pipeline generates high-quality samples comprising original and modified code, along with natural language instructions in different styles and verbosity. Today's code LMs come bundled with strong abilities, such as code generation and instruction following, which should not be lost due to fine-tuning. To ensure this, we propose a novel adaptation algorithm, SeleKT, that (a) leverages a dense gradient-based step to identify the weights that are most important for code editing, and (b) does a sparse projection onto the base model to avoid overfitting. Using our approach, we obtain a new series of models NextCoder (adapted from QwenCoder-2.5) that achieves strong results on five code-editing benchmarks, outperforming comparable size models and even several larger ones. We show the generality of our approach on two model families (DeepSeekCoder and QwenCoder), compare against other fine-tuning approaches, and demonstrate robustness by showing retention of code generation and general problem-solving abilities post adaptation. We opensource the models, synthetic dataset, and implementation at https://aka.ms/nextcoder.

代码生成模型适配合成数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。