让大模型自己判断答案是否在记忆中，减少50%检索次数
Let your LLM generate a few tokens and you will reduce the need for retrieval
- 用大模型当裁判生成训练数据，判断答案是否已在记忆中
- 在特定数据集上可减少超50%的搜索与重排序步骤
- 仅需约2万样本训练，适合想优化检索效率的研究者
本文研究大语言模型（LLM）能否高效判断答案是否已存储于其参数化记忆中。通过构建一个以LLM为裁判的判别器来计算IK（I Know）分数，发现该方法在检索增强生成（RAG）场景下表现优异，准确率达80%。该方法可显著减少某些数据集上超过50%的搜索和重排序步骤。我们还引入了IK分数，用于通过分类任务刻画数据集特性。有趣的是，加入响应词元作为输入后，仅需约20,000个训练样本即可达到良好性能。本工作的核心是使用教师模型（即大模型作为裁判）生成训练数据。我们还评估了不同教师类型下的鲁棒性，包括基于字符串的方法和大模型，后者效果更优。
原文摘要 · Abstract (English)
In this paper, we investigate how efficiently large language models (LLM) can be trained to check whether an answer is already stored in their parametric memory. We distill an LLM-as-a-judge to compute the IK (I Know) score. We found that this method is particularly beneficial in the context of retrieval-assisted augmented generation (RAG), with a respectable accuracy of 80%. It enables a significant reduction (more than 50%) in the number of search and reranking steps required for certain data sets. We have also introduced the IK score, which serves as a useful tool for characterising datasets by facilitating the classification task. Interestingly, through the inclusion of response tokens as input, our results suggest that only about 20,000 training samples are required to achieve good performance. The central element of this work is the use of a teacher model - the LLM as a judge - to generate training data. We also assess the robustness of the IK classifier by evaluating it with various types of teachers, including both string-based methods and LLMs, with the latter providing better results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。