arXiv:2605.16608cs.LGcs.CL2026-05

不使用MRL也能让文本嵌入抗截断,除非截断超80%。

To MRL or not to MRL: Text Embeddings are Robust to Truncation Without Matryoshka Learning, Except In Heavy Truncation Scenarios

  • 对比随机截断与MRL训练模型的嵌入性能
  • 截断低于80%时,非MRL模型表现更优或相当
  • 适合需轻量级嵌入且不需极端压缩的研究者

Matryoshka Representation Learning(MRL)是一种广泛采用的方法,用于训练文本编码器,使其在不同尺寸下提供有效表示,只需在训练时预设尺寸并截断向量。近期研究表明,随机截断文本嵌入对下游任务影响极小,除非向量尺寸减少至少70%。然而,此前未有研究比较随机截断与MRL的效果。本文通过将MRL所用截断策略应用于训练有无MRL的模型,在多个模型和下游任务上进行评估。结果表明,除非极端截断(即尺寸减少至少80%),否则非MRL模型的截断嵌入表现可媲美甚至优于MRL模型。这表明截断鲁棒性未必来自MRL,是否投入额外训练成本取决于是否需要重度压缩。代码已开源供复现。

原文摘要 · Abstract (English)

Matryoshka Representation Learning (MRL) is a widely adopted approach for training text encoders so they provide useful text representations at various sizes, available by simply truncating the resulting vectors at sizes pre-determined at training time. Recent works have shown that randomly truncating text embeddings has minimal impact in downstream performance unless vectors are reduced in size by at least 70%, suggesting that embeddings are already robust to truncation without the use of MRL. However, no prior work has compared random truncation to MRL, so it is unclear how the two methods compare as effective embedding reduction methods. In this paper, we study this by applying the same truncation used by MRL to models trained with and without MRL. Our results across several models and downstream tasks show that, unless heavily truncating embeddings (i.e. reducing their size by at least 80%), truncated embeddings of non-MRL models are competitive with, and often outperform models trained with MRL. This suggests that truncation robustness may not necessarily come from MRL, and that the choice of spending the additional training cost of MRL depends on whether heavy truncation is desired. We make our code available for reproduction.

文本嵌入截断鲁棒性MRL

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。