从零训练14亿参数模型，实现跨语言文本嵌入新基准
Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings
- 从头训练模型并引入新闻与多语言数据，缩小预训练差距
- 设计软掩码机制平滑过渡因果与双向训练，提升表征能力
- 动态难负样本挖掘提升效果，中文版达最新最佳性能
大型语言模型（LLM）在文本嵌入任务中表现优异。以往工作通常使用LoRA微调现有LLM，受限于预训练数据与嵌入模型间的差异。本文提出Conan-embedding-v2，一个1.4B参数的全新大模型，从头训练并微调为文本嵌入器。首先，通过加入新闻数据和多语言对增强预训练，弥合数据差距；在此基础上，构建跨语言检索数据集，促进多语言嵌入融合。其次，针对LLM采用因果掩码与词元级损失、嵌入模型采用双向掩码与句级损失的训练差异，提出软掩码机制，逐步过渡两种掩码方式，使模型学习更全面的表示。进一步设计动态硬负样本挖掘方法，在训练过程中持续引入更难的负例。该模型仅约1.4B参数，即在Massive Text Embedding Benchmark（MTEB）及中文MTEB（2025年5月19日更新）上达到当前最优性能。
原文摘要 · Abstract (English)
Large language models (LLMs) have recently demonstrated excellent performance in text embedding tasks. Previous work usually use LoRA to fine-tune existing LLMs, which are limited by the data and training gap between LLMs and embedding models. In this work, we introduce Conan-embedding-v2, a new 1.4B-parameter LLM trained from scratch and fine-tuned as a text embedder. First, we add news data and multilingual pairs for LLM pretraining to bridge the data gap. Based on this, we propose a cross-lingual retrieval dataset that enables the LLM to better integrate embeddings across different languages. Second, whereas LLMs use a causal mask with token-level loss, embedding models use a bidirectional mask with sentence-level loss. This training gap makes full fine-tuning less effective than LoRA. We introduce a soft-masking mechanism to gradually transition between these two types of masks, enabling the model to learn more comprehensive representations. Based on this, we propose a dynamic hard negative mining method that exposes the model to more difficult negative examples throughout the training process. Being intuitive and effective, with only approximately 1.4B parameters, Conan-embedding-v2 achieves SOTA performance on both the Massive Text Embedding Benchmark (MTEB) and Chinese MTEB (May 19, 2025).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。