用实体关系图和模型感知采样提升文档检索嵌入精度
Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Sampling
- 基于实体关系图进行中断微调，引导模型关注事实而非风格
- 采用模型感知对比采样，平衡难易负样本分布，提升检索准确率
- 在长文本检索中实现90.86%的rank@1准确率，优于现有方法
本文提出APEX-Embedding-7B（Advanced Processing for Epistemic eXtraction），一个70亿参数的解码器仅模型，专为文档检索增强生成（RAG）任务设计。该方法采用两种训练技术：(1) 使用结构化实体关系图作为输入进行预收敛中断微调，引导模型注意力偏向事实内容而非语义风格，从而提升纯文本表现；(2) 模型感知对比采样，依据基础模型能力生成均衡分布的难例与易例负样本组合。联合方法显著提升性能，在评估中使纯文本查询/文档对检索的rank@1准确率达到90.86%，较次优模型提升6.26%；同时平均减少37.71%的训练输入上下文长度。本模型在长上下文文档检索任务中树立了新基准。
原文摘要 · Abstract (English)
In this paper we present APEX-Embedding-7B (Advanced Processing for Epistemic eXtraction), a 7-billion parameter decoder-only text Feature Extraction Model, specifically designed for Document Retrieval-Augmented Generation (RAG) tasks. Our approach employs two training techniques that yield an emergent improvement in factual focus: (1) Pre-convergence interrupted fine-tuning using Structured Entity Relationship Maps as training data input: designed to shift the model's attention and create a bias towards factual content rather than semantic style - this enhances plain text performance despite not being directly trained for it; and (2) Model-Aware Contrastive Sampling, creating a balanced and evenly distributed collation map of hard and soft negatives directly informed by the base model's competency. This combined methodology yields significant improvements, enhancing plain text query/document pair retrieval to achieve an absolute rank@1 accuracy of 90.86% (an increase of 6.26% compared to the next leading model) in our evaluation, and reducing training data input context size by an average of 37.71% compared to plain text for both queries and document texts. Based on our evaluations, our model establishes a new state-of-the-art standard in text feature extraction for longer context document retrieval tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。