首个通用稀疏专家混合文本嵌入模型，兼顾性能与效率。
Training Sparse Mixture Of Experts Text Embedding Models
- 采用稀疏专家混合架构，动态激活部分参数提升推理效率。
- 在单语和多语言基准上超越同规模模型，媲美双倍大小模型。
- 开源完整训练流程，适合关注高效文本嵌入的研究与应用者。
基于Transformer的文本嵌入模型通过增加参数量，在MIRACL和BEIR等基准上取得了性能提升。然而，这种扩展方式带来显著部署挑战，包括推理延迟和内存占用增加，尤其在检索增强生成（RAG）应用中，大模型的高内存需求限制了数据集摄入能力，高延迟直接影响查询性能。尽管因果语言模型已通过专家混合（MoE）架构解决类似效率问题，但该方法尚未成功应用于通用文本嵌入场景。本文提出Nomic Embed v2，首个通用型MoE文本嵌入模型。其在同参数规模下优于现有模型，且性能媲美参数量两倍大的模型。我们开源全部代码、模型及评估数据，确保训练流程可复现，详见 https://github.com/nomic-ai/contrastors。
原文摘要 · Abstract (English)
Transformer-based text embedding models have improved their performance on benchmarks like MIRACL and BEIR by increasing their parameter counts. However, this scaling approach introduces significant deployment challenges, including increased inference latency and memory usage. These challenges are particularly severe in retrieval-augmented generation (RAG) applications, where large models' increased memory requirements constrain dataset ingestion capacity, and their higher latency directly impacts query-time performance. While causal language models have addressed similar efficiency challenges using Mixture of Experts (MoE) architectures, this approach hasn't been successfully adapted to the general text embedding setting. In this paper, we introduce Nomic Embed v2, the first general purpose MoE text embedding model. Our model outperforms models in the same parameter class on both monolingual and multilingual benchmarks while also maintaining competitive performance with models twice its size. We open-source all code, models, and evaluation data to ensure full reproducibility of our training pipeline at \href{https://github.com/nomic-ai/contrastors}{https://github.com/nomic-ai/contrastors}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。