用合成数据训练代码嵌入模型，提升通用代码检索能力
Towards A Generalist Code Embedding Model Based On Massive Data Synthesis
- 基于多样性、可靠性、可用性原则生成大规模合成数据集
- 在16个任务中显著超越现有模型，跨领域泛化能力强
- 适合需要高效代码检索的开发者和研究者使用
由于检索增强生成（RAG）在软件开发中的广泛应用，代码嵌入模型受到越来越多关注。这些模型需捕捉代码特有的丰富语义关系，其特性与文本差异显著。然而，现有模型受限于高质量训练数据稀缺。本文提出 extbf{CodeR}（– Code Retrieval），一种面向通用代码检索的先进嵌入模型。CodeR 的优异性能建立在 CodeR-Pile 上，这是一个通过创新数据合成流程构建的大规模合成数据集，遵循 DRU（多样性、可靠性、可用性）原则。为提升训练效率，我们设计了 Annealing 课程学习策略，实现异构数据源间的有效知识迁移。我们在 16 个多样化的代码检索任务上评估 CodeR，结果显著优于现有基线，并展现出强大的跨领域泛化能力。代码与预训练模型已开源，以促进该关键领域的进一步研究：https://github.com/FlagOpen/FlagEmbedding/tree/master/research/BGE_Coder。
原文摘要 · Abstract (English)
Code embedding models attract increasing attention due to the widespread popularity of retrieval-augmented generation (RAG) in software development. These models are expected to capture the rich semantic relationships inherent to code, which differ significantly from those found in text. However, existing models remain severely limited due to the scarcity of high-quality training data. In this work, we introduce \textbf{CodeR} (\underline{Code} \underline{R}etrieval), a state-of-the-art embedding model for general-purpose code retrieval. The superior performance of CodeR is built upon CodeR-Pile, a large-scale synthetic dataset constructed under the DRU (Diversity, Reliability, Usability) principle via a novel data synthesis pipeline. To optimize training effectiveness, we propose Annealing, a curriculum learning strategy that enables effective knowledge transfer across heterogeneous sources of data. We evaluate CodeR based on 16 diverse code retrieval tasks, where it significantly outperforms existing baselines and exhibits strong out-of-domain generalization performance. We have publicly released our code and the well-trained model to facilitate further research in this critical area. https://github.com/FlagOpen/FlagEmbedding/tree/master/research/BGE_Coder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。