用轻量开源模型评估RAG幻觉，成本低且可解释。
Towards Lighter and Robust Evaluation for Retrieval Augmented Generation
- 用小型量化模型替代GPT-4做RAG生成质量评估。
- 输出连续得分，能衡量答案正确性与忠实度。
- 适合关注RAG可靠性与低成本评估的研究者。
大语言模型促使我们从生成视角看待更多自然语言处理任务，同时通过检索增强生成（RAG）框架提供了新的信息获取方式。尽管自回归模型已有显著提升，生成答案中的幻觉问题仍持续存在。目前主流方法依赖商业大模型如GPT-4进行评估，但成本高且不透明。为此，本文提出利用开源权重模型评估RAG幻觉的可行性。我们开发了一种轻量级方法，采用更小、已量化的大模型，提供一种可访问、可解释的连续评分机制，用于衡量生成答案在正确性与忠实度方面的表现。该评分可用于评估决策可靠性，并探索阈值以构建一种替代人类判断相关性的新AUC指标。
原文摘要 · Abstract (English)
Large Language Models are prompting us to view more NLP tasks from a generative perspective. At the same time, they offer a new way of accessing information, mainly through the RAG framework. While there have been notable improvements for the autoregressive models, overcoming hallucination in the generated answers remains a continuous problem. A standard solution is to use commercial LLMs, such as GPT4, to evaluate these algorithms. However, such frameworks are expensive and not very transparent. Therefore, we propose a study which demonstrates the interest of open-weight models for evaluating RAG hallucination. We develop a lightweight approach using smaller, quantized LLMs to provide an accessible and interpretable metric that gives continuous scores for the generated answer with respect to their correctness and faithfulness. This score allows us to question decisions' reliability and explore thresholds to develop a new AUC metric as an alternative to correlation with human judgment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。