用一致性检验提升大模型对检索增强生成的评估准确性
Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models
- 通过多维度判断并检验一致性，筛选可靠评价
- 在多个RAG模型和数据集上显著提升评估准确率
- 适合需要高质量自动评估的模型训练与评测场景
检索增强生成（RAG）能有效缓解大语言模型的幻觉问题。然而，现有自动化评估指标无法公平评价RAG模型在训练与评估阶段的输出。基于大语言模型的判断模型虽具高评价潜力，但对评估提示高度敏感，导致对RAG输出的判断不一致。本文提出Judge-Consistency（ConsJudge）方法，通过让大模型基于不同判断维度生成多组评价，利用判断一致性筛选出可信的接受与拒绝样本，用于直接偏好优化（DPO）训练。实验表明，ConsJudge在多种RAG模型与数据集上均能提供更准确的评估结果。进一步分析显示，ConsJudge生成的判断与优秀大模型具有高一致性。代码已开源：https://github.com/OpenBMB/ConsJudge。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has proven its effectiveness in alleviating hallucinations for Large Language Models (LLMs). However, existing automated evaluation metrics cannot fairly evaluate the outputs generated by RAG models during training and evaluation. LLM-based judgment models provide the potential to produce high-quality judgments, but they are highly sensitive to evaluation prompts, leading to inconsistencies when judging the output of RAG models. This paper introduces the Judge-Consistency (ConsJudge) method, which aims to enhance LLMs to generate more accurate evaluations for RAG models. Specifically, ConsJudge prompts LLMs to generate different judgments based on various combinations of judgment dimensions, utilize the judge-consistency to evaluate these judgments and select the accepted and rejected judgments for DPO training. Our experiments show that ConsJudge can effectively provide more accurate judgments for optimizing RAG models across various RAG models and datasets. Further analysis reveals that judgments generated by ConsJudge have a high agreement with the superior LLM. All codes are available at https://github.com/OpenBMB/ConsJudge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。