先进编码器在稀疏检索中表现差,因词汇表设计导致语义匹配失效。
Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps
- 通过词汇粗粒度化缓解语义冗余,提升模型泛化能力
- 新方法使ModernBERT在BEIR上达52.4 nDCG,提升4.7分
- 适用于各类模型,尤其适合低资源或专用领域
尽管现代基础模型如ModernBERT在稠密检索中显著优于旧架构,但在学习型稀疏检索(LSR)中反而落后于老旧的BERT-base。我们发现根本原因在于「词汇缺口」:现代分词器采用原始、大小写敏感的词汇表,将单一语义单元映射为冗余表面形式,浪费模型容量于形态噪声,阻碍词汇匹配。我们通过理论框架证明,适当词汇粗粒度化可降低假设类复杂度,从而收紧泛化界,前提是保持语义完整。为此提出『词汇迁移(VT)』,一种无需修改模型的通用框架,以极低计算成本将先进编码器迁移到适配稀疏检索的规范化词汇表。VT结合新颖的『语义初始化』(基于空间拓扑保留几何结构)与『激活潜力校准(APC)』机制,避免标准微调中的死神经元和密集坍缩问题。实验证明,VT普适有效:使ModernBERT在BEIR基准上达到52.4 nDCG(+4.7),恢复失败模型如RoBERTa-large,且可无缝迁移至无推理架构与特定领域。结果表明性能差距非架构缺陷,而是可解的词汇不匹配问题。代码与模型已公开。
原文摘要 · Abstract (English)
While advanced foundation models like ModernBERT significantly outperform older architectures in dense retrieval, they surprisingly lag behind the aging BERT-base baseline in learned sparse retrieval (LSR). We identify the root cause as the \textit{Vocabulary Gap}: modern tokenizers utilize raw, case-sensitive vocabularies designed for lossless reconstruction, which map single semantic units to redundant surface forms, wasting model capacity on morphological noise and hindering lexical matching. We formalize this intuition through a theoretical framework, demonstrating that appropriate vocabulary coarse-graining can tighten the generalization bounds by reducing complexity of the hypothesis class, provided that semantic integrity is preserved. To resolve this, we propose \textbf{Vocabulary Transfer (VT)}, a model-agnostic framework that migrates advanced encoders to sparse-friendly, normalized vocabularies with minimal computational cost. VT utilizes a novel \textbf{Semantic Initialization} via spatial topology to preserve geometric structure and an \textbf{Activation Potential Calibration (APC)} mechanism to align pre-trained manifolds with sparsity constraints, preventing the dead neuron and dense collapse observed in standard fine-tuning. Empirically, VT is universally effective: it enables ModernBERT to achieve state-of-the-art performance on the BEIR benchmark (\textbf{52.4} nDCG, a \textbf{+4.7} improvement), resuscitates failing models like RoBERTa-large, and generalizes seamlessly to inference-free architectures and specialized domains. These results confirm that the performance lag is not an architectural deficiency but a solvable vocabulary mismatch. We've released our code and models.\footnote{https://anonymous.4open.science/r/vocab-transfer/. All details included.}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。