用神经检索实现代码语义搜索,解决命名混淆难题
MediaWiki Code2Code Search: Neural Retrieval for the Semantic Discovery of Open-Source Software Entities
- 采用拆分构建架构,离线用GPU索引,线上纯CPU服务
- 1.29百万实体索引,延迟仅1.85秒,内存占用168.6MB
- 适合需要语义级代码发现的开发者和开源研究者
大规模代码生态中的代码搜索常受查询与实现间词汇鸿沟困扰,且传统信息检索(IR)低延迟与深度学习(DL)高精度之间存在权衡。我们提出MediaWiki Code2Code Search,一个用于语义代码到代码发现的神经检索系统。该系统对2,500多个MediaWiki仓库中的129万条结构化实体(函数、类型、模板)进行索引,支持基于计算意图而非表面词元的检索。采用拆分构建架构,将耗GPU的离线索引与纯CPU的在线服务解耦;使用FAISS IVF-PQ索引,仅占168.6 MB,相比平坦float32基线减少96.6%;在普通硬件上实现1.85秒中位查询延迟,满足Wikimedia Toolforge的6 GiB内存限制。在27个查询的基准测试中,性能优于BM25基线,P@10达0.87(基线为0.64),严格匹配下分别为0.52与0.34。在名称混淆任务中提升尤为显著。系统已通过https://code2codesearch.toolforge.org 开放,采用Apache 2.0许可,并提供开放的RESTful API。
原文摘要 · Abstract (English)
Code search in large-scale ecosystems is often hindered by the lexical gap between user queries and implementation details, alongside the trade-off between the low latency of traditional Information Retrieval (IR) and the precision of Deep Learning (DL). We present MediaWiki Code2Code Search, a neural retrieval system for semantic code-to-code discovery. By indexing 1.29 million structural entities (functions, types, and templates) across 2,500+ MediaWiki repositories, our system enables retrieval based on computational intent rather than surface tokens. We employ a split-build architecture, decoupling GPU-intensive offline indexing from a CPU-only serving layer; our FAISS IVF-PQ index occupies 168.6 MB: a 96.6\% reduction compared to a flat float32 baseline, and achieves a median query latency of 1.85 seconds on commodity hardware, satisfying the 6 GiB RAM constraint of Wikimedia Toolforge. Our evaluation across a 27-query benchmark demonstrates superior performance over the BM25 baseline, achieving a P@10 of 0.87 compared to 0.64 (0.52 versus 0.34 for strict matching). Gains are most pronounced in name-obfuscated tasks where lexical methods fail. The system is available at https://code2codesearch.toolforge.org under the Apache 2.0 licence and provides an open RESTful API.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。