用大模型生成训练数据，让检索器性能突破现有水平。
Syntriever: How to Train Your Retriever with Synthetic Data from LLMs
- 用链式思考生成相关与看似无关的文本，提升检索质量。
- 在多个数据集上，nDCG@K 指标达到当前最优水平。
- 适合想提升检索系统效果的研究者与工程师使用。
大型语言模型（LLMs）推动了诸多AI应用的发展。近期有研究尝试将大模型的丰富知识蒸馏到信息检索系统中，但这些方法多依赖于模型输出的概率，而最新黑盒大模型无法提供该信息。为此，我们提出Syntriever，一种利用黑盒大模型生成合成数据来训练检索器的框架。该框架包含两个阶段：首先，在蒸馏阶段，基于给定查询，通过链式思考生成相关及看似无关的段落和增强查询；随后，大模型自验证生成数据是否存在幻觉，再以嵌入聚类损失训练检索器。其次，在对齐阶段，通过一种称为部分Plackett-Luce排序的偏好建模方法，学习大模型的偏好，并引入正则化防止模型偏离蒸馏阶段训练结果。实验表明，Syntriever在多个领域基准数据集上的nDCG@$K$指标达到当前最优水平。代码已开源于https://github.com/kmswin1/Syntriever。
原文摘要 · Abstract (English)
LLMs have boosted progress in many AI applications. Recently, there were attempts to distill the vast knowledge of LLMs into information retrieval systems. Those distillation methods mostly use output probabilities of LLMs which are unavailable in the latest black-box LLMs. We propose Syntriever, a training framework for retrievers using synthetic data from black-box LLMs. Syntriever consists of two stages. Firstly in the distillation stage, we synthesize relevant and plausibly irrelevant passages and augmented queries using chain-of-thoughts for the given queries. LLM is asked to self-verify the synthetic data for possible hallucinations, after which retrievers are trained with a loss designed to cluster the embeddings of relevant passages. Secondly in the alignment stage, we align the retriever with the preferences of LLMs. We propose a preference modeling called partial Plackett-Luce ranking to learn LLM preferences with regularization which prevents the model from deviating excessively from that trained in the distillation stage. Experiments show that Syntriever achieves state-of-the-art performances on benchmark datasets from various domains in nDCG@$K$. The code is available at \href{https://github.com/kmswin1/Syntriever}{https://github.com/kmswin1/Syntriever}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。