通过分层压缩提升大模型嵌入的效率与精度
SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression
- 采用分层嵌入学习缓解训练梯度波动
- 压缩至256维后在BEIR上性能提升1.1~2.7点
- 适合需要高效部署大模型嵌入的场景
大型语言模型生成高维嵌入，虽蕴含丰富语义与句法信息，但带来计算复杂度与存储压力，阻碍实际部署。为此，我们提出新型训练框架序列式套娃嵌入压缩（SMEC）。该框架引入序列式套娃表示学习（SMRL）以缓解训练中的梯度方差，自适应维度选择（ADS）模块减少降维过程中的信息损失，并可选跨批次记忆（S-XBM）模块增强高低维嵌入间的无监督学习。在图像、文本及多模态数据集上的实验表明，SMEC在显著降低维度的同时保持性能。例如，在BEIR数据集上，相比Matryoshka-Adaptor与Search-Adaptor模型，压缩至256维的LLM2Vec嵌入分别提升1.1点与2.7点。
原文摘要 · Abstract (English)
Large language models (LLMs) generate high-dimensional embeddings that capture rich semantic and syntactic information. However, high-dimensional embeddings exacerbate computational complexity and storage requirements, thereby hindering practical deployment. To address these challenges, we propose a novel training framework named Sequential Matryoshka Embedding Compression (SMEC). This framework introduces the Sequential Matryoshka Representation Learning(SMRL) method to mitigate gradient variance during training, the Adaptive Dimension Selection (ADS) module to reduce information degradation during dimension pruning, and the Selectable Cross-batch Memory (S-XBM) module to enhance unsupervised learning between high- and low-dimensional embeddings. Experiments on image, text, and multimodal datasets demonstrate that SMEC achieves significant dimensionality reduction while maintaining performance. For instance, on the BEIR dataset, our approach improves the performance of compressed LLM2Vec embeddings (256 dimensions) by 1.1 points and 2.7 points compared to the Matryoshka-Adaptor and Search-Adaptor models, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。