arXiv:2606.18811cs.IRcs.AI2026-06

调校语言模型头的尺度,让大模型更稳定地用于稀疏检索。

Rescaling MLM-Head for Neural Sparse Retrieval

  • 用常数因子在初始化时缩放MLM头输出,解决大模型训练崩溃问题。
  • 在多个数据集上,修正后的大模型性能超越传统BERT-SPLADE基线。
  • 适合想用更强预训练模型做稀疏检索的研究者和工程师。

基于学习的稀疏检索(LSR)模型如SPLADE通常采用BERT风格的掩码语言模型作为编码器骨干。直观预期是替换为更强的预训练编码器可提升检索效果。然而我们发现,在标准SPLADE训练流程下,具有较大MLM头L2范数的编码器反而导致性能下降甚至训练崩溃。这归因于MLM头尺度与SPLADE表示构造方式之间的不匹配:SPLADE直接使用MLM头输出构建稀疏词汇表示,查询-文档相关性通过未归一化的点积计算。过大的MLM头尺度会放大稀疏激活,扭曲匹配分数,破坏对比学习稳定性。为此,我们提出一种初始化阶段的简单校正方法——在SPLADE训练前对MLM头投影乘以常数因子。该零成本调整不改变模型结构或训练目标,显著提升训练稳定性。在跨域与领域内检索任务中,修正后的ModernBERT和Ettin等大范数模型表现优异,部分场景下超越经典BERT-SPLADE。结果表明,适配预训练模型到LSR的瓶颈并非编码器容量,而是用于构建稀疏表示的MLM头尺度需精准校准。

原文摘要 · Abstract (English)

Learned sparse retrieval (LSR) models such as SPLADE have traditionally used BERT-style masked language models as backbone encoders. A natural expectation is that replacing BERT with stronger pretrained encoders should improve retrieval effectiveness. However, we find that under standard SPLADE training recipes, backbones with large MLM-head L2 norms can suffer performance degradation and even training collapse under standard SPLADE training recipes. We identify this failure as a scale mismatch in the MLM head: SPLADE directly uses MLM-head outputs to construct sparse lexical representations, and query-document relevance is computed by an unnormalized dot product over these representations. As a result, an inflated MLM-head scale can amplify sparse activations, distort matching scores, and destabilize contrastive training under common training settings. To address this issue, we introduce a simple initialization-time correction that rescales the MLM-head projection by a constant factor before SPLADE training. This zero-cost adjustment improves training stability without modifying the model architecture or training objective. Across both in-domain and out-of-domain retrieval benchmarks, this simple correction substantially improves large-norm backbones such as ModernBERT and Ettin, turning unstable training runs into competitive sparse retrievers. In several settings, the corrected models further match or surpass the classic BERT-SPLADE baseline. These findings suggest that the bottleneck in adapting pretrained encoders to LSR is not encoder capacity alone, but the calibration of the MLM-head scale used to construct sparse lexical representations.

稀疏检索模型校准BERT改进训练稳定

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。