无需微调,统一框架生成强通用文本嵌入。
LGAI-EMBEDDING-Preview Technical Report
- 用提示词引导大模型生成上下文感知嵌入
- 软标签与自适应难例挖掘提升语义区分度
- 在41项任务中表现顶尖,适合多场景应用
本报告提出一种基于指令的统一框架,用于学习适用于信息检索(IR)与非IR任务的通用文本嵌入。基于Mistral-7B解码器模型,方法结合上下文学习、软监督和自适应硬负例挖掘,在不进行任务特定微调的情况下生成上下文感知嵌入。通过结构化指令与少样本示例引导模型完成分类、语义相似性、聚类与重排序等任务。为增强语义区分能力,引入软标签框架,利用高性能稠密检索器与重排序器蒸馏出连续相关性评分作为细粒度监督信号。同时提出基于自适应边距的硬负例挖掘,依据负例与正例的相似度过滤语义模糊样本,提升训练稳定性和检索鲁棒性。在新提出的MTEB(英语版,v2)基准上评估,涵盖7个类别共41项任务。结果表明,该方法具有强大泛化能力,Borda得分位列前列,优于多个更大或全微调基线。验证了指令提示、软监督与自适应采样结合在可扩展高质量嵌入生成中的有效性。
原文摘要 · Abstract (English)
This report presents a unified instruction-based framework for learning generalized text embeddings optimized for both information retrieval (IR) and non-IR tasks. Built upon a decoder-only large language model (Mistral-7B), our approach combines in-context learning, soft supervision, and adaptive hard-negative mining to generate context-aware embeddings without task-specific fine-tuning. Structured instructions and few-shot examples are used to guide the model across diverse tasks, enabling strong performance on classification, semantic similarity, clustering, and reranking benchmarks. To improve semantic discrimination, we employ a soft labeling framework where continuous relevance scores, distilled from a high-performance dense retriever and reranker, serve as fine-grained supervision signals. In addition, we introduce adaptive margin-based hard-negative mining, which filters out semantically ambiguous negatives based on their similarity to positive examples, thereby enhancing training stability and retrieval robustness. Our model is evaluated on the newly introduced MTEB (English, v2) benchmark, covering 41 tasks across seven categories. Results show that our method achieves strong generalization and ranks among the top-performing models by Borda score, outperforming several larger or fully fine-tuned baselines. These findings highlight the effectiveness of combining in-context prompting, soft supervision, and adaptive sampling for scalable, high-quality embedding generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。