用智能筛选提升代码大模型训练效率，10K数据超92K全量数据
Efficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data Selection
- 用参数化模型筛选代码数据，保证分布一致与多样性
- 仅用1万样本，性能超越9.2万样本基线，提升2.4%和2.3%
- 适合追求高效训练的代码生成研究者与工程团队
大语言模型在代码生成与程序理解方面取得显著进展，加速了软件工程发展。现有方法主要依赖海量数据提升性能，但常忽视数据质量，降低训练效率。为此，我们提出一种基于参数化模型的代码数据选择方法，优化所选子集的分布一致性与多样性，确保高质量数据。实验表明，仅使用10,000个样本，该方法在HumanEval上比92,000个样本的全量基线提升2.4%，在MBPP上提升2.3%，优于其他采样策略，在性能与效率上均表现更优。这证明该方法能有效提升模型性能并显著降低计算成本。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have significantly improved code generation and program comprehension, accelerating the evolution of software engineering. Current methods primarily enhance model performance by leveraging vast amounts of data, focusing on data quantity while often overlooking data quality, thereby reducing training efficiency. To address this, we introduce an approach that utilizes a parametric model for code data selection, aimed at improving both training efficiency and model performance. Our method optimizes the parametric model to ensure distribution consistency and diversity within the selected subset, guaranteeing high-quality data. Experimental results demonstrate that using only 10K samples, our method achieves gains of 2.4% (HumanEval) and 2.3% (MBPP) over 92K full-sampled baseline, outperforming other sampling approaches in both performance and efficiency. This underscores that our method effectively boosts model performance while significantly reducing computational costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。