让大模型每步推理都可靠，防止错误传播。
PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation
- 用生成式评分模型同时评估逻辑和证据，每步判断是否合理。
- 在5个问答数据集上，平均精确率与召回率均最优。
- 适合需要严谨多步推理的科研、医疗等高要求场景。
检索增强生成通过引入外部知识提升大模型表现，但多跳推理易受错误传播影响——早期检索失败会干扰后续步骤。传统基于结果的优化仅奖励最终答案，忽视中间环节的错误。现有过程驱动方法虽引入步骤级信号，但仍以最终答案为标准评分，可能误奖因错误检索却巧合得正确答案的情况。步骤级监督需在每一步评估逻辑合理性与证据支撑性。本文提出PRO-STEP：训练一个生成式偏好模型（PRM）同时衡量这两维；利用该模型引导值树搜索构建对比偏好对，区分有效步骤与缺陷步骤；并采用步骤级直接偏好优化（DPO）训练策略。在单跳与多跳问答数据集上的实验表明，PRO-STEP在五个基准测试中均达到最高平均准确率（EM）与F1值。代码、模型及训练数据已开源至https://github.com/keemminnke/PRO-Step。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation enhances Large Language Models by grounding responses in external knowledge, but multi-hop reasoning remains vulnerable to error propagation, where early retrieval failures confound subsequent steps. Standard outcome-based optimization only rewards the final answer, leaving intermediate retrieval and reasoning errors undetected. While existing process-based methods introduce step-level signals, they still score each step against the final answer, rewarding spurious successes where flawed retrieval coincidentally produces the correct answer. Step-level supervision in RAG requires evaluating both logical validity and evidential grounding at each step. We introduce PRO-STEP: we train a generative PRM that evaluates both dimensions, employ PRM-guided value tree search to construct preference pairs contrasting valid steps against flawed ones, and optimize the policy via step-level Direct Preference Optimization. Experiments on single and multi-hop QA datasets demonstrate that PRO-STEP achieves the best average EM and F1 across five benchmarks. Code, models, and training data are publicly available at https://github.com/keemminnke/PRO-Step.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。