用q-对数改进BM25，显著提升代码检索准确率
Improving BM25 Code Retrieval Under Fixed Generic Tokenization: Adaptive q-Log Odds as a Drop-In BM25 Fix
- 用q-对数替换BM25的对数变换，增强函数名区分能力
- 在CodeSearchNet上NDCG@10提升89.3%，达0.4874
- 无需改分词器，可直接替换现有BM25索引使用
在检索增强编程中，相关文件常因缺失而无法召回。在固定通用分词条件下，传统BM25的对数型RSJ-IDF会弱化函数间区分特征。本文将罗伯逊-斯帕尔克-琼斯对数替换为q-对数，在q=1时还原原BM25，q<1时为Box-Cox变换（λ=1−q）。在包含18.2万文档的CoIR CodeSearchNet Go数据集上，最优调参后NDCG@10从0.2575升至0.4874（绝对提升+0.2299，相对提升+89.3%），10,000次配对自助抽样中无符号反转（p ≤ 10^-4）。该效果在不同编程语言间呈梯度变化，而在BEIR文本数据上几乎无效。通过词频唯一性密度可估算全局q值，当BM25已最优时该值趋近1。索引时间仅需一次稀疏得分矩阵遍历，查询延迟不变。分词器消融实验表明，若分词器本身具备标识符感知能力，则q-IDF带来的增益大幅减弱。
原文摘要 · Abstract (English)
In retrieval-augmented coding, failures often begin when the relevant file is absent from the retrieved context. Under frozen generic tokenization, where a BM25 index has been built by a search system whose analyzer the practitioner does not control, this failure is routine: BM25's logarithmic RSJ-odds IDF under-separates the identifier tail that distinguishes one function from another. We replace the outer logarithm of the Robertson-Spärck-Jones odds with a q-logarithm. At q=1 the transform recovers BM25 exactly by L'Hôpital's rule, and for q<1 it is a Box-Cox transform of the RSJ odds with lambda = 1-q. On CoIR CodeSearchNet Go (182K documents), oracle-tuned NDCG@10 rises from 0.2575 to 0.4874 (absolute +0.2299; +89.3% relative; zero sign reversals in 10,000 paired-bootstrap resamples, reported as p <= 10^-4). The effect is graded across code languages and is near-zero on BEIR text. A one-parameter closed form estimates a corpus-level q from hapax density and stays near q=1 on corpora where BM25 is already optimal. The index-time cost is a single pass over the sparse score matrix and query latency is unchanged. A tokenizer ablation shows that identifier-aware tokenization largely removes the incremental gain from q-IDF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。