用不确定性得分引导大模型自我校准，提升问答准确率。
SGIC: A Self-Guided Iterative Calibration Framework for RAG
- 基于不确定性得分动态评估文档相关性与回答置信度。
- 多轮迭代优化，使闭源与开源大模型性能显著提升。
- 自建训练集，让模型更高效利用不确定信息。
检索增强生成（RAG）研究主要聚焦于从候选文档中检索有用信息，但多数方法忽视了大语言模型（LLMs）的校准能力，而这些能力正源于其强大的上下文推理能力。本文表明，向LLMs提供特定提示可显著提升其校准效果，尤其在多轮校准中更为明显。我们提出SGIC：自引导迭代校准框架，利用不确定性得分作为工具。首先，该框架计算不确定性得分，以判断每份文档与查询的相关性以及模型输出回答的置信度；随后，通过迭代重评得分，并结合先前响应结果进行校准优化。此外，我们设计了一种创新的迭代式自校准训练集构建方法，使LLMs能更高效地利用不确定性得分捕捉关键信息并提高回答准确性。实验显示，该框架在闭源与开源大模型上均取得显著性能提升。
原文摘要 · Abstract (English)
Recent research in retrieval-augmented generation (RAG) has concentrated on retrieving useful information from candidate documents. However, numerous methodologies frequently neglect the calibration capabilities of large language models (LLMs), which capitalize on their robust in-context reasoning prowess. This work illustrates that providing LLMs with specific cues substantially improves their calibration efficacy, especially in multi-round calibrations. We present a new SGIC: Self-Guided Iterative Calibration Framework that employs uncertainty scores as a tool. Initially, this framework calculates uncertainty scores to determine both the relevance of each document to the query and the confidence level in the responses produced by the LLMs. Subsequently, it reevaluates these scores iteratively, amalgamating them with prior responses to refine calibration. Furthermore, we introduce an innovative approach for constructing an iterative self-calibration training set, which optimizes LLMs to efficiently harness uncertainty scores for capturing critical information and enhancing response accuracy. Our proposed framework significantly improves performance on both closed-source and open-weight LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。