用大模型生成多层次难例,提升文本嵌入精度
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings
- 用大模型生成不同相似度的难样本,实现由粗到精的学习
- 在MTEB上达到顶尖性能,优于现有合成数据方法
- 无需增加复杂度,通过锚点标记加权提升效果
文本嵌入模型在自然语言处理中至关重要,能将语义信息编码为密集向量。这类模型通常通过三元组(查询、正样本、负样本)进行对比学习优化,其中负样本对区分细微语义差异起关键作用。本文提出多粒度难例合成框架(MGH),利用大语言模型生成与查询具有不同相似度的多样化负样本,支持从粗到细的课程学习策略,使嵌入模型逐步掌握更精细的语义表征。同时,提出锚点标记感知池化(ATA)方法,根据大模型中的聚合模式为锚点标记分配更高权重,提升文本嵌入精度而不增加模型复杂度。在MTEB基准上的全面实验表明,该方法在仅使用合成数据时即达到顶尖性能,且在结合公开检索数据集后表现更优。
原文摘要 · Abstract (English)
Text embedding models are essential for various natural language processing tasks, enabling the effective encoding of semantic information into dense vector representations. These models are typically optimized using triplets of (query, positive, negative) data pairs for contrastive learning, where the negative samples play a critical role in enhancing the model's ability to discern subtle semantic distinctions. In this work, we introduce a Multi-Granularity Hard-negative (MGH) synthesis framework that leverages large language models (LLMs) to generate diverse negative samples with varying levels of similarity with the query. This approach facilitates a coarse-to-fine curriculum learning strategy during supervised training, allowing the embedding model to progressively learn more nuanced semantic representations. Meanwhile, we propose an Anchor Token Aware (ATA) pooling method that assigns higher weights to anchor tokens based on aggregation patterns observed in LLMs, improving text embedding accuracy without increasing model complexity. Comprehensive experiments on the MTEB benchmark demonstrate that our methods achieve state-of-the-art performance, surpassing existing synthesis strategies both with synthetic data and when combined with public retrieval datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。