通过排序标签捕捉代码负样本间的细微差异,提升代码搜索效果
OASIS: Order-Augmented Strategy for Improved Code Search
- 引入顺序增强策略,利用负样本间的相似性排序训练模型
- 在多个基准测试中超越现有最优模型,显著提升代码搜索精度
- 适合需要精准代码语义理解的研究者与工程团队
代码嵌入能捕捉代码的语义表示,在各类代码相关大语言模型应用中至关重要,如代码搜索。以往训练主要依赖对比正负样本对(自然语言-代码)的InfoNCE损失。然而由于代码上下文稀疏,仅关注正负样本间的主要差异,难以捕捉深层语义细节。为此,我们提出一种新的顺序增强策略(OASIS),利用基于顺序的相似性标签,使模型学习负样本之间的细微相似性差异。大量基准测试表明,我们的OASIS模型显著优于仅关注主次差异的现有最先进模型,证明了利用负样本间排序标签进行有效代码嵌入训练的价值。
原文摘要 · Abstract (English)
Code embeddings capture the semantic representations of code and are crucial for various code-related large language model (LLM) applications, such as code search. Previous training primarily relies on optimizing the InfoNCE loss by comparing positive natural language (NL)-code pairs with in-batch negatives. However, due to the sparse nature of code contexts, training solely by comparing the major differences between positive and negative pairs may fail to capture deeper semantic nuances. To address this issue, we propose a novel order-augmented strategy for improved code search (OASIS). It leverages order-based similarity labels to train models to capture subtle differences in similarity among negative pairs. Extensive benchmark evaluations demonstrate that our OASIS model significantly outperforms previous state-of-the-art models focusing solely on major positive-negative differences. It underscores the value of exploiting subtle differences among negative pairs with order labels for effective code embedding training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。