解决检索模型持续学习时的嵌入漂移问题,无需重索引即可保持旧数据可用性。
Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models
- 通过查询嵌入漂移补偿,将新模型的查询投影到旧嵌入空间。
- 在不重索引情况下,显著降低旧任务遗忘率,提升持续学习性能。
- 适用于需要动态更新但无法重新索引的大规模语义搜索系统。
文本嵌入模型支持语义搜索,广泛应用于检索增强生成等NLP任务。然而,现有研究多基于静态训练数据,难以应对动态场景中持续新增数据的情况。传统检索方法将大量文档编码为低维嵌入并存入索引库,查询时通过相似度匹配返回最相关文档。当使用新数据更新嵌入模型时,由于模型变化导致旧索引嵌入与新模型不兼容,直接使用旧索引效果下降。虽重索引可恢复兼容性,但计算成本高。本文建立大规模持续学习基准,持续在新数据集上训练密集检索嵌入模型,发现旧任务存在显著嵌入漂移导致遗忘。为此,我们对查询和文档嵌入同时进行知识蒸馏,并提出新颖的查询漂移补偿机制,在检索阶段将新模型的查询嵌入投影至旧嵌入空间,实现与旧索引的兼容,有效缓解遗忘。实验表明该方法无需重索引即显著提升性能。代码已开源:https://github.com/dipamgoswami/QDC。
原文摘要 · Abstract (English)
Text embedding models enable semantic search, powering several NLP applications like Retrieval Augmented Generation by efficient information retrieval (IR). However, text embedding models are commonly studied in scenarios where the training data is static, thus limiting its applications to dynamic scenarios where new training data emerges over time. IR methods generally encode a huge corpus of documents to low-dimensional embeddings and store them in a database index. During retrieval, a semantic search over the corpus is performed and the document whose embedding is most similar to the query embedding is returned. When updating an embedding model with new training data, using the already indexed corpus is suboptimal due to the non-compatibility issue, since the model which was used to obtain the embeddings of the corpus has changed. While re-indexing of old corpus documents using the updated model enables compatibility, it requires much higher computation and time. Thus, it is critical to study how the already indexed corpus can still be effectively used without the need of re-indexing. In this work, we establish a continual learning benchmark with large-scale datasets and continually train dense retrieval embedding models on query-document pairs from new datasets in each task and observe forgetting on old tasks due to significant drift of embeddings. We employ embedding distillation on both query and document embeddings to maintain stability and propose a novel query drift compensation method during retrieval to project new model query embeddings to the old embedding space. This enables compatibility with previously indexed corpus embeddings extracted using the old model and thus reduces the forgetting. We show that the proposed method significantly improves performance without any re-indexing. Code is available at https://github.com/dipamgoswami/QDC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。