arXiv:2411.17299cs.IRcs.CL2024-11被引 9

2D马特罗什卡训练让模型同时适应多层多维嵌入,提升文本检索效果。

2D Matryoshka Training for Information Retrieval

  • 在不同层数和维度上联合训练编码器,统一优化多粒度嵌入表示。
  • 在STS和检索任务中优于传统马特罗什卡训练,但不及单独训练特定维度的模型。
  • 适合需要高效多尺度嵌入的检索场景,尤其零样本和监督检索任务。

2D马特罗什卡训练是一种先进的嵌入表示训练方法,可使编码器模型在多种层-维度配置下同时进行训练。该方法在语义文本相似性(STS)任务中表现出比传统训练方式更高的有效性,特别是在使用子层嵌入时。尽管如此,两种已发布的实现版本存在差异,导致与基线模型比较结果不一致。本可复现性研究对两种版本的2D马特罗什卡训练在STS任务中进行了实现与评估,并将分析扩展至检索任务。结果表明,两种版本在子维度上均优于传统马特罗什卡训练和全尺寸模型训练,但未能超越针对特定子层与子维度单独训练的模型。这些结果在监督(MSMARCO)和零样本(BEIR)检索设置中均有良好泛化能力。进一步分析显示,引入全维度损失并覆盖更广的目标维度范围,能更好地适配检索任务;而固定文档编码器为全模型输出等直观做法则未带来改进。复现代码见 https://github.com/ielab/2DMSE-Reproduce。

原文摘要 · Abstract (English)

2D Matryoshka Training is an advanced embedding representation training approach designed to train an encoder model simultaneously across various layer-dimension setups. This method has demonstrated higher effectiveness in Semantic Text Similarity (STS) tasks over traditional training approaches when using sub-layers for embeddings. Despite its success, discrepancies exist between two published implementations, leading to varied comparative results with baseline models. In this reproducibility study, we implement and evaluate both versions of 2D Matryoshka Training on STS tasks and extend our analysis to retrieval tasks. Our findings indicate that while both versions achieve higher effectiveness than traditional Matryoshka training on sub-dimensions, and traditional full-sized model training approaches, they do not outperform models trained separately on specific sub-layer and sub-dimension setups. Moreover, these results generalize well to retrieval tasks, both in supervised (MSMARCO) and zero-shot (BEIR) settings. Further explorations of different loss computations reveals more suitable implementations for retrieval tasks, such as incorporating full-dimension loss and training on a broader range of target dimensions. Conversely, some intuitive approaches, such as fixing document encoders to full model outputs, do not yield improvements. Our reproduction code is available at https://github.com/ielab/2DMSE-Reproduce.

嵌入训练文本检索多尺度表示可复现性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。