开源的80亿参数多语言嵌入模型,性能领先且支持自定义指令。
Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks
- 采用1610万条查询-文档对训练,含770万真实数据与840万合成数据。
- 在多语言任务中表现卓越,尤其在低资源语言和跨语言场景下领先。
- 公开权重与消融实验,支持定制指令,适合需要灵活嵌入的开发者。
我们推出 llama-embed-nemotron-8b,一个开放权重的文本嵌入模型,截至2025年10月21日,在多语言大规模文本嵌入基准(MMTEB)上达到顶尖水平。现有模型虽性能优异,但训练数据或方法常未完全公开。为此,我们构建了全开源模型,公开模型权重、详细消融研究,并计划分享整理后的训练数据集。该模型在检索、分类及语义文本相似性(STS)等主要任务中均表现优异,尤其在低资源语言和跨语言场景中优势显著。其卓越性能源于1610万条查询-文档对的数据混合,其中770万来自公开数据集,840万由多种开源大模型生成的合成数据。关键贡献包括对对比损失实现、合成数据生成策略及模型融合影响的系统性消融分析。llama-embed-nemotron-8b为指令感知模型,支持用户自定义指令以提升特定场景表现,结合顶级性能、广泛适用性与用户可定制性,可作为通用文本嵌入解决方案。
原文摘要 · Abstract (English)
We introduce llama-embed-nemotron-8b, an open-weights text embedding model that achieves state-of-the-art performance on the Multilingual Massive Text Embedding Benchmark (MMTEB) leaderboard as of October 21, 2025. While recent models show strong performance, their training data or methodologies are often not fully disclosed. We aim to address this by developing a fully open-source model, publicly releasing its weights and detailed ablation studies, and planning to share the curated training datasets. Our model demonstrates superior performance across all major embedding tasks -- including retrieval, classification and semantic textual similarity (STS) -- and excels in challenging multilingual scenarios, such as low-resource languages and cross-lingual setups. This state-of-the-art performance is driven by a novel data mix of 16.1 million query-document pairs, split between 7.7 million samples from public datasets and 8.4 million synthetically generated examples from various open-weight LLMs. One of our key contributions is a detailed ablation study analyzing core design choices, including a comparison of contrastive loss implementations, an evaluation of synthetic data generation (SDG) strategies, and the impact of model merging. The llama-embed-nemotron-8b is an instruction-aware model, supporting user-defined instructions to enhance performance for specific use-cases. This combination of top-tier performance, broad applicability, and user-driven flexibility enables it to serve as a universal text embedding solution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。