不依赖预训练，纯用公开数据训练的ColBERT模型表现超越现有顶尖模型。
ColBERT-Zero: To Pre-train Or Not To Pre-train ColBERT models
- 直接在公开数据上全量预训练多向量模型，跳过传统知识蒸馏流程。
- ColBERT-Zero在相同规模下超越GTE-ModernColBERT和其基础模型。
- 统一预训练与微调设置对模型复用至关重要，适合检索系统研究者。
当前最先进的多向量模型通常基于强单向量模型的小规模知识蒸馏训练，利用这些模型的大规模预训练。本文研究多向量模型的预训练策略，发现大规模多向量预训练可显著提升模型性能。值得注意的是，仅使用公开数据训练的完全预训练模型ColBERT-Zero，优于GTE-ModernColBERT及其基础模型GTE-ModernBERT（后者使用封闭且更强的数据），在该规模模型中创下新基准。我们还发现，仅进行小规模知识蒸馏不足以接近全预训练效果，但若先加入有监督训练步骤，则可大幅缩小差距，同时跳过最耗时的无监督阶段。最后，我们强调在复用现有模型时，需保持微调与预训练设置的一致性。为支持后续研究，我们发布多种检查点及训练代码。
原文摘要 · Abstract (English)
Current state-of-the-art multi-vector models are obtained through a small Knowledge Distillation (KD) training step on top of strong single-vector models, leveraging the large-scale pre-training of these models. In this paper, we study the pre-training of multi-vector models and show that large-scale multi-vector pre-training yields much stronger multi-vector models. Notably, a fully ColBERT-pre-trained model, ColBERT-Zero, trained only on public data, outperforms GTE-ModernColBERT as well as its base model, GTE-ModernBERT, which leverages closed and much stronger data, setting new state-of-the-art for model this size. We also find that, although performing only a small KD step is not enough to achieve results close to full pre-training, adding a supervised step beforehand allows to achieve much closer performance while skipping the most costly unsupervised phase. Finally, we find that aligning the fine-tuning and pre-training setups is crucial when repurposing existing models. To enable exploration of our results, we release various checkpoints as well as code used to train them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。