用多阶段蒸馏让小模型逼近大模型，提升检索效率。
Jasper and Stella: distillation of SOTA embedding models
- 通过三类损失函数，让小模型蒸馏多个大模型知识。
- 20亿参数的小模型在MTEB上获第三名，平均得分71.54。
- 结合马特罗什卡学习降维，适合部署于实际应用。
密集检索是问答系统与检索增强生成等深度学习应用的核心组件，依赖嵌入模型将文本转换为向量。当前在大规模文本嵌入基准（MTEB）表现优异的嵌入模型通常参数量大、向量维度高，难以部署于真实场景。为此，我们提出一种多阶段蒸馏框架，使小型学生模型通过三种精心设计的损失函数，从多个大型教师模型中学习。同时，利用马特罗什卡表示学习（MRL）有效降低学生模型的向量维度。所提出的名为Jasper的学生模型（基于Stella架构，含20亿参数），在2024年12月24日的MTEB排行榜上位列第3，56个数据集上平均得分为71.54。模型与数据已发布于Hugging Face Hub，训练代码开源。
原文摘要 · Abstract (English)
A crucial component in many deep learning applications, such as Frequently Asked Questions (FAQ) and Retrieval-Augmented Generation (RAG), is dense retrieval. In this process, embedding models transform raw text into numerical vectors. However, the embedding models that currently excel on text embedding benchmarks, like the Massive Text Embedding Benchmark (MTEB), often have numerous parameters and high vector dimensionality. This poses challenges for their application in real-world scenarios. To address this issue, we propose a novel multi-stage distillation framework that enables a smaller student embedding model to distill multiple larger teacher embedding models through three carefully designed losses. Meanwhile, we utilize Matryoshka Representation Learning (MRL) to reduce the vector dimensionality of the student embedding model effectively. Our student model named Jasper with 2 billion parameters, built upon the Stella embedding model, obtained the No.3 position on the MTEB leaderboard (as of December 24, 2024), achieving an average 71.54 score across 56 datasets. We have released the model and data on the Hugging Face Hub (https://huggingface.co/infgrad/jasper_en_vision_language_v1) (https://huggingface.co/datasets/infgrad/jasper_text_distill_dataset), and the training codes are available in this project repository (https://github.com/NLPJCL/RAG-Retrieval).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。