清理代码指令数据泄漏问题，用新策略训练出更优的代码大模型
How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data
- 从指令复杂度、回复质量、多样性三方面筛选高质量数据
- 使用更少数据实现新最优性能，超越现有主流模型
- 适合关注代码生成与数据质量的开发者和研究者
近期对构建更好代码指令数据的兴趣日益增长。然而我们发现，使用这些数据训练的代码模型在HumanEval上表现优异，但在LiveCodeBench等其他基准上表现更差。深入分析发现，许多数据集存在严重数据泄漏问题。清理大部分泄漏数据后，一些知名高质量数据集的表现反而下降。这揭示了新挑战：如何真正识别高质量代码指令数据。为此，我们提出一种高效的代码数据剪枝策略，基于指令复杂度、响应质量与指令多样性三个维度筛选样本。基于所选数据，我们推出了从LLaMA3微调而来的XCoder系列模型。实验表明，XCoder在更少训练数据下达到新SOTA性能，验证了数据策略的有效性。此外，我们对数据组成进行了全面分析，发现不同构建方法的数据具有不同特征，为未来代码大模型发展提供新洞见。模型与数据已开源于https://github.com/banksy23/XCoder。
原文摘要 · Abstract (English)
Recently, there has been a growing interest in studying how to construct better code instruction tuning data. However, we observe Code models trained with these datasets exhibit high performance on HumanEval but perform worse on other benchmarks such as LiveCodeBench. Upon further investigation, we find that many datasets suffer from severe data leakage. After cleaning up most of the leaked data, some well-known high-quality datasets perform poorly. This discovery reveals a new challenge: identifying which dataset genuinely qualify as high-quality code instruction data. To address this, we propose an efficient code data pruning strategy for selecting good samples. Our approach is based on three dimensions: instruction complexity, response quality, and instruction diversity. Based on our selected data, we present XCoder, a family of models finetuned from LLaMA3. Our experiments show XCoder achieves new state-of-the-art performance using fewer training data, which verify the effectiveness of our data strategy. Moreover, we perform a comprehensive analysis on the data composition and find existing code datasets have different characteristics according to their construction methods, which provide new insights for future code LLMs. Our models and dataset are released in https://github.com/banksy23/XCoder
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。