arXiv:2607.27178cs.CLcs.IR2026-07被引 1

开源训练全流程,实现多语言长文本检索新纪录。

DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search

  • 用6.65亿对英文数据重建预训练集,结合硬负样本挖掘微调。
  • 在BEIR上达57.22的nDCG@10,刷新同规模模型纪录。
  • 晚交互模型更适应未见语言,证明细粒度匹配提升泛化能力。

当前顶尖检索模型依赖封闭数据,造成可复现性差距。本文提出一个开源端到端训练方案,研究英语监督在多语言检索中的迁移效果。我们从34个公开来源的14亿对数据中重构并清洗出6.65亿对英文对比预训练样本,并构建了188万对带挖掘硬负例的监督微调数据。训练得到两个149M参数模型:DenseOn(单向量密集模型)和LateOn(ColBERT风格晚交互模型),在BEIR基准上分别取得56.20和57.22的平均nDCG@10,创下该规模下的新纪录。随后将验证后的英文数据翻译成八种语言,生成28亿对跨语言样本,训练出基于mmBERT-base的mDenseOn和mLateOn(307M参数)。尽管共享骨干、数据与目标,两者表征行为不同:密集模型在英语及翻译语言上表现强,但在非translate-train支持的语言上退化;而晚交互模型在未见语言和书写系统上更具泛化能力。这表明词元级匹配使translate-train从目标语言扩展策略转变为多语言泛化方法。所有模型、数据集与训练代码均已开源。

原文摘要 · Abstract (English)

State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training retrieval models and study how English supervision transfers to multilingual retrieval through translate-train. We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources and build 1.88M supervised fine-tuning pairs with mined hard negatives. Training yields two 149M-parameter models: DenseOn, a single-vector dense model, and LateOn, a ColBERT-style late-interaction model. They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class. We then translate the validated English data into eight languages, yielding 2.8B pairs with cross-lingual samples, and train mDenseOn and mLateOn, two 307M-parameter models built on mmBERT-base. Despite sharing their backbone, data, and objectives, their representations behave differently: the dense model is strong on English and translated languages but degrades outside translate-train support, whereas the late-interaction model generalizes better to unseen languages and scripts. This suggests that token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe. We publicly release the models, datasets, and training code.

检索模型多语言开放数据晚交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。