不修改模型,直接优化向量嵌入,让检索更准更快。
NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval
- 不依赖参数,直接调整数据向量以提升检索精度
- 在9个数据集上平均提升NDCG@10超10%,速度比微调快200倍
- 适合需要快速高精度检索的RAG系统开发者
基于预训练模型生成的稠密向量嵌入进行k近邻搜索(k-NN检索)是文本与图像检索以及检索增强生成(RAG)管道中的主流方法。实际应用中,开发者常通过微调嵌入来提升特定数据集和查询负载下的准确性。现有方法要么微调预训练模型本身,要么更高效但牺牲准确率地训练适配器模型转换输出。本文提出NUDGE,一种新型非参数化嵌入微调方法,显著优于两类现有方法。NUDGE直接修改数据记录的嵌入,以最大化k-NN检索的准确性。我们进行了理论与实验分析,证明尽管原始问题是NP难的,但受约束的变体可高效求解,且变化幅度小,避免破坏预训练语义。在五个预训练模型与九个标准文本/图像检索数据集上的实验表明,NUDGE运行仅需几分钟,多数情况下使NDCG@10提升超过10%;平均而言,其准确率提升为微调模型的3.3倍、为适配器的4.3倍,运行速度分别快200倍和3倍。
原文摘要 · Abstract (English)
$k$-Nearest Neighbor search on dense vector embeddings ($k$-NN retrieval) from pre-trained embedding models is the predominant retrieval method for text and images, as well as Retrieval-Augmented Generation (RAG) pipelines. In practice, application developers often fine-tune the embeddings to improve their accuracy on the dataset and query workload in hand. Existing approaches either fine-tune the pre-trained model itself or, more efficiently, but at the cost of accuracy, train adaptor models to transform the output of the pre-trained model. We present NUDGE, a family of novel non-parametric embedding fine-tuning approaches that are significantly more accurate and efficient than both sets of existing approaches. NUDGE directly modifies the embeddings of data records to maximize the accuracy of $k$-NN retrieval. We present a thorough theoretical and experimental study of NUDGE's non-parametric approach. We show that even though the underlying problem is NP-Hard, constrained variations can be solved efficiently. These constraints additionally ensure that the changes to the embeddings are modest, avoiding large distortions to the semantics learned during pre-training. In experiments across five pre-trained models and nine standard text and image retrieval datasets, NUDGE runs in minutes and often improves NDCG@10 by more than 10% over existing fine-tuning methods. On average, NUDGE provides 3.3x and 4.3x higher increase in accuracy and runs 200x and 3x faster, respectively, over fine-tuning the pre-trained model and training adaptors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。