arXiv:2605.19568cs.CL2026-05KDD被引 1

一个可适配多种资源需求的多语言文本编码器,提升工业检索效率。

m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder

  • 通过多粒度嵌入维度联合预训练,实现单模型多规格输出。
  • 在Bing-Click数据集上超越现有最佳模型,兼顾精度与效率。
  • 适合需要灵活部署的工业级搜索与广告系统使用。

嵌入模型在工业信息检索系统(如搜索和广告)中至关重要,但现有预训练模型通常架构固定、嵌入维度单一,难以适应不同部署场景下的资源约束。常见做法是用大模型部分参数初始化小任务,但因预训练与下游任务不匹配,效果受限。为此,我们提出m3BERT:一种现代、多语言、马特罗什卡结构的双向编码器,采用新型联合优化策略,在多个嵌入维度和Transformer层间同步优化表示能力,使单一模型能根据资源与精度目标灵活调整。模型采用三阶段预训练:单语种预训练、多语言适配以服务多元用户群体,以及在大规模网络语料上的持续预训练,显著提升商业检索实用性。在Bing-Click这一大规模工业检索数据集上,m3BERT显著优于当前最优嵌入模型,验证其作为资源感知型工业检索基础模型的高效性。公开数据集上的实验也证实了多粒度马特罗什卡预训练策略的通用有效性。

原文摘要 · Abstract (English)

Embedding models are pivotal in industrial information retrieval systems like search and advertising. However, existing pretrained models often exhibit fixed architectures and embedding dimensionalities, posing significant challenges when adapting them to diverse deployment scenarios with varying business-driven constraints. A common practice involves fine-tuning with partial parameter initialization from larger pretrained models for resource-constrained tasks. This method is often suboptimal as the misalignment between pretraining and downstream usage prevents full realization of pretraining benefits. To address this limitation, we introduce m3BERT: a Modern, Multi-lingual, Matryoshka Bidirectional Encoder, which features a novel pretraining strategy that jointly optimizes representations across both transformer layers and multiple embedding dimensions. This enables a single model to be tailored to varied resource and accuracy targets while maintaining consistency with pretraining. Incorporating recent architectural improvements, m3BERT uses a three-stage pretraining: monolingual pretraining, multilingual adaptation to serve diverse user bases, and crucial continual pretraining on a massive web domain corpus to enhance utility in commercial retrieval. m3BERT significantly outperforms state-of-the-art embedding models in Bing-Click, a large-scale industrial retrieval dataset, showcasing its practical versatility as an efficient foundation for resource-aware industrial retrieval systems. Further experiments on public datasets also confirm the general effectiveness of our multigranular Matryoshka pretraining strategy.

多语言嵌入模型工业检索马特罗什卡

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。