用强化学习训练结构化批评者，精准诊断并修正RAG生成错误。
CRITIC-R1: Learning Structured Critics for Retrieval-Augmented Generation
- 将RAG错误分为判别、位置、推理和修复四类，构建结构化诊断框架
- 在5个问答数据集上优于强基线，显著降低幻觉与推理错误
- 适合需要高可靠性生成的场景，如医疗、法律等知识密集型应用
检索增强生成（RAG）通过引入外部证据提升知识密集型问答表现，但仍存在幻觉和细微推理错误。现有方法引入外部批评者进行优化，但反馈粗粒度、结构弱、干预过激，导致修正结果噪声大且不可靠。为此，我们提出CRITIC-R1，一个基于强化学习的结构化批评框架，将RAG批评建模为显式的错误诊断问题。该框架将常见RAG错误划分为判决、错误位置、推理分析和修复生成四个诊断维度。为学习这些能力，设计两种奖励函数：保守判断对齐（CJA）鼓励校准的高层判断，缓解过度干预；诊断质量对齐（DQA）通过门控奖励提升细粒度诊断反馈。使用基于GRPO的强化学习，结合外部大模型教师提供的过程级监督进行训练。在五个QA基准上的实验表明，CRITIC-R1持续优于强基线。源代码已公开于https://anonymous.4open.science/r/critic-r1-FCB0。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) improves knowledge-intensive question answering by incorporating external evidence. However, existing RAG methods still suffer from hallucinations and subtle reasoning errors. Recent studies introduce external critics to refine RAG outputs, yet they often provide coarse-grained and weakly structured feedback, exhibit over-aggressive intervention, and lead to noisy and unreliable refinement, limiting their effectiveness for correction. To tackle these issues, we propose CRITIC-R1, a structured critic framework that formulates and learns RAG critique as an explicit error diagnosis problem using reinforcement learning (RL). Our framework categorizes common RAG errors into multiple diagnostic dimensions, including verdict, error location, reasoning analysis, and fix generation. To learn these capabilities, we design two reward functions: Conservative Judgement Alignment (CJA) first encourages calibrated high-level judgements while mitigating the over-aggressive phenomenon, whereas Diagnostic Quality Alignment (DQA) further improves fine-grained diagnostic feedback through gated rewards. We train the critic model using GRPO-based RL with process-level supervision collected from external LLM teacher models. Experiments across five QA benchmarks show that CRITIC-R1 consistently improves answer quality over strong RAG baselines. Our source code is available at https://anonymous.4open.science/r/critic-r1-FCB0
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。