arXiv:2510.12327cs.IRcs.AI2025-10被引 2

改进ColBERT的投影层,能显著提升检索性能。

Simple Projection Variants Improve ColBERT Performance

  • 用更复杂的前馈网络替代原始线性投影层。
  • 最佳变体使多个基准上NDCG@10提升超2点。
  • 残差连接和中间维度扩展是关键因素。

多向量稠密检索方法如ColBERT使用单层线性投影降低向量维度。本文研究了MaxSim算子对多向量模型训练梯度流的影响,发现该线性投影存在固有但非关键的局限性。通过设计并系统评估替代投影块,我们表明优化后的投影可显著提升ColBERT下游性能。实验显示,多种投影变体均优于原始线性投影,最优方案在多个领域检索基准上平均提升超过2 NDCG@10点。消融分析揭示,中间维度扩大与残差连接尤为重要。此外,该效果在不同随机种子下一致,验证了替换ColBERT线性层是一种稳健、可直接部署的升级方案。

原文摘要 · Abstract (English)

Multi-vector dense retrieval methods like ColBERT systematically use a single-layer linear projection to reduce the dimensionality of individual vectors. In this study, we explore the implications of the MaxSim operator on the gradient flows of the training of multi-vector models and show that such a simple linear projection has inherent, if non-critical, limitations in this setting. We then discuss the theoretical improvements that could result from replacing this single-layer projection with well-studied alternative feedforward linear networks (FFN), such as deeper, non-linear FFN blocks, GLU blocks, and skip-connections, could alleviate these limitations. Through the design and systematic evaluation of alternate projection blocks, we show that better-designed final projections positively impact the downstream performance of ColBERT models. We highlight that many projection variants outperform the original linear projections, with the best-performing variants increasing average performance on a range of retrieval benchmarks across domains by over 2 NDCG@10 points. We then conduct further exploration on the individual parameters of these projections block in order to understand what drives this empirical performance, highlighting the particular importance of upscaled intermediate projections and residual connections. As part of these ablation studies, we show that numerous suboptimal projection variants still outperform the traditional single-layer projection across multiple benchmarks, confirming our hypothesis. Finally, we observe that this effect is consistent across random seeds, further confirming that replacing the linear layer of ColBERT models is a robust, drop-in upgrade.

检索深度学习向量投影ColBERT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。