用语义困惑度衡量检索信息对生成的增益效果。
SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction
- 提出语义困惑度（SePer）衡量检索内容对大模型内部信念的修正程度。
- 实验显示SePer与人类偏好高度一致，且评估效率高于传统指标。
- 适合关注检索质量评估、RAG系统优化的研究者使用。
大型语言模型通过引入外部检索知识提升了生成性能，这一过程称为检索增强生成（RAG）。然而，现有研究或联合评估检索与生成环节，难以分离检索的独立贡献；或使用NDCG等传统指标评估检索器，无法反映检索在整体生成过程中的真实价值。为此，本文提出一种基于RAG框架的自动评估方法——语义困惑度（SePer），通过量化大模型在检索后语义困惑度的下降程度，衡量检索信息带来的信息增益。实验表明，SePer不仅与人类偏好高度一致，且在多种RAG场景下提供更精确、高效的检索效用评估。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated improved generation performance by incorporating externally retrieved knowledge, a process known as retrieval-augmented generation (RAG). Despite the potential of this approach, existing studies evaluate RAG effectiveness by 1) assessing retrieval and generation components jointly, which obscures retrieval's distinct contribution, or 2) examining retrievers using traditional metrics such as NDCG, which creates a gap in understanding retrieval's true utility in the overall generation process. To address the above limitations, in this work, we introduce an automatic evaluation method that measures retrieval quality through the lens of information gain within the RAG framework. Specifically, we propose Semantic Perplexity (SePer), a metric that captures the LLM's internal belief about the correctness of the retrieved information. We quantify the utility of retrieval by the extent to which it reduces semantic perplexity post-retrieval. Extensive experiments demonstrate that SePer not only aligns closely with human preferences but also offers a more precise and efficient evaluation of retrieval utility across diverse RAG scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。