arXiv:2605.04615cs.SEcs.AI2026-05被引 1

构建首个覆盖代码搜索全流程的多任务评测基准,解决数据污染与标签噪声问题。

Beyond Retrieval: A Multitask Benchmark and Model for Code Search

论文配图:Beyond Retrieval: A Multitask Benchmark and Model for Code Search
图 1 · 摘自论文原文
  • 提出多任务基准CoREB,涵盖检索与重排序全链条
  • 短关键词查询使所有模型性能暴跌至nDCG@10接近零
  • 首个在三类任务中均实现稳定提升的可微分重排序模型

代码搜索通常仅被评估为单阶段检索,但实际系统依赖包含重排序和开发者风格查询的完整流水线。现有基准存在数据污染、标签噪声及退化二元相关性问题。本文提出 extsc{CoREB},一个无污染、多任务的代码检索与重排序基准,包含五个编程语言的反事实改写LiveCodeBench问题,采用分级相关性标注并限时发布。我们在三类任务(文本到代码、代码到文本、代码到代码)上评估了十一种嵌入模型与五种重排序器。实验发现:代码专用嵌入在代码到代码检索中表现优异(约2倍于通用编码器),但无模型能通吃所有任务;最接近真实开发者的短关键词查询使所有模型nDCG@10接近零;现成重排序器呈现任务不对称性,代码到代码任务最高相差12分,且无基线在所有任务上获得正向增益;我们提出的微调模型 extsc{CoREB-Reranker}是首个在三类任务中均实现一致提升的模型。数据与模型已开源。

原文摘要 · Abstract (English)

Code search has usually been evaluated as first-stage retrieval, even though production systems rely on broader pipelines with reranking and developer-style queries. Existing benchmarks also suffer from data contamination, label noise, and degenerate binary relevance. In this paper, we introduce \textsc{CoREB}, a contamination-limited, multitask \underline{co}de \underline{r}etrieval and r\underline{e}ranking \underline{b}enchmark, together with a fine-tuned code reranker, that goes beyond retrieval to cover the full code search pipeline. \textsc{CoREB} is built from counterfactually rewritten LiveCodeBench problems in five programming languages and delivered as timed releases with graded relevance judgments. We benchmark eleven embedding models and five rerankers across three tasks: text-to-code, code-to-text, and code-to-code. Our experiments reveal that: \circone code-specialised embeddings dominate code-to-code retrieval (${\sim}2{\times}$ over general encoders), yet no single model wins all three tasks; \circtwo short keyword queries, the format closest to real developer search, collapse every model to near-zero nDCG@10; \circthree off-the-shelf rerankers are task-asymmetric, with a 12-point swing on code-to-code and no baseline net-positive across all tasks; \circfour our fine-tuned \textsc{CoREB-Reranker} is the first to achieve consistent gains across all three tasks. The data and model are released.

代码搜索多任务评测重排序基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。