用规则推理实现高效可信的RAG评估，无需大量参数。
RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning
- 将评估任务转为规则引导的推理，支持单次生成完整判断。
- 相比人类判断相关性最强，且仅需基线1/100参数量。
- 零人工标注生成评分参考，结果解释性强，适合可信AI验证。
可靠的评估对部署可信的检索增强生成（RAG）系统至关重要。然而，当前基于大模型的评估框架主要依赖资源密集型模型进行复杂多阶段提示，未能充分挖掘模型推理能力，且计算成本高。本文提出RAG-Zeval（RAG-零评估器），一种端到端框架，将忠实性与正确性评估建模为规则引导的推理任务。通过强化学习训练评估器，使小型模型能在一次遍历中生成全面且合理的评估及详细解释。我们引入基于排序的结果奖励机制，采用偏好判断而非绝对分数，以应对精确点对点奖励信号获取困难的问题。为此，我们通过零人工标注生成质量可控的响应作为排名参考。实验表明，RAG-Zeval在与人类判断的相关性上表现最优，优于依赖10至100倍参数的大模型基线。该方法还展现出更强的可解释性，适用于RAG系统的可信评估。
原文摘要 · Abstract (English)
Robust evaluation is critical for deploying trustworthy retrieval-augmented generation (RAG) systems. However, current LLM-based evaluation frameworks predominantly rely on directly prompting resource-intensive models with complex multi-stage prompts, underutilizing models' reasoning capabilities and introducing significant computational cost. In this paper, we present RAG-Zeval (RAG-Zero Evaluator), a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task. Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments with detailed explanation in one-pass. We introduce a ranking-based outcome reward mechanism, using preference judgments rather than absolute scores, to address the challenge of obtaining precise pointwise reward signals. To this end, we synthesize the ranking references by generating quality-controlled responses with zero human annotation. Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments and outperforming baselines that rely on LLMs with 10-100 times more parameters. Our approach also exhibits superior interpretability in response evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。