让大模型生成时主动评估检索文档质量,提升回答可靠性。
OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAG
- 引入文档相关性、排序和查询表现评分作为显式质量特征
- 在5个数据集上优于基线,对噪声上下文更鲁棒
- 可灵活适配不同后训练目标和外部指标
大型语言模型(LLMs)在众多下游任务中表现出色,包括基于检索增强生成(RAG)的场景。生成内容的质量高度依赖于检索信息的相关性以及模型内部处理机制的有效性。通常假设检索到的信息与问题相关,但实际相关性和有用性会因问题和文档集合而异。因此,在生成过程中考虑检索信息的相关性至关重要。本文提出OpenDecoder,一种利用显式评估结果作为质量指标特征来指导生成的新方法。我们考虑三种显式评估信息:相关性得分、排序得分和查询性能预测(QPP)得分。在五个基准数据集上的实验表明,OpenDecoder不仅有效,且对不同噪声上下文具有更强的鲁棒性,显著优于多种基线方法。更重要的是,该范式可灵活集成至任何后训练目标,并兼容各类外部指标。
原文摘要 · Abstract (English)
The development of large language models (LLMs) has achieved superior performance in a range of downstream tasks, including LLM-based retrieval-augmented generation (RAG). The quality of generated content heavily relies on the usefulness of the retrieved information and the capacity of LLMs' internal information processing mechanism to incorporate it in answer generation. It is generally assumed that the retrieved information is relevant to the question. However, the retrieved information may have a variable degree of relevance and usefulness, depending on the question and the document collection. It is important to take into account the relevance of the retrieved information in answer generation. In this paper, we propose OpenDecoder, a new approach that leverages explicit evaluation of the retrieved information as quality indicator features for generation. We aim to build a RAG model that is more robust to varying levels of noisy context. Three types of explicit evaluation information are considered: relevance score, ranking score, and QPP (query performance prediction) score. The experimental results on five benchmark datasets demonstrate the effectiveness and better robustness of OpenDecoder by outperforming various baseline methods. Importantly, this paradigm is flexible to be integrated with the post-training of LLMs for any purposes and incorporated with any type of external indicators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。