让大模型推理时实时获得领域知识反馈,提升医疗问答准确率。
Process Reward Agents for Steering Knowledge-Intensive Reasoning

- 推理过程中动态生成步骤奖励,基于外部知识即时评估每步正确性。
- 在MedQA上达81.9%准确率,比现有方法高出显著,且适用于0.5B~8B参数模型。
- 无需更新模型即可提升性能,适合部署新推理框架的复杂领域应用。
知识密集型推理仍具挑战,因中间步骤难以局部验证:与数学或代码不同,判断步骤正确性需整合大量外部知识源。细微错误可能在推理链中持续传播且无法被发现。以往工作提出过程奖励模型(PRM),包括检索增强型变体,但这些方法为事后评分,无法融入动态推理流程。本文提出过程奖励代理(PRA),一种推理时的在线、逐步奖励机制,为冻结策略模型提供领域感知奖励。相比先前检索增强的PRM,PRA支持基于搜索的解码,在每一步对候选轨迹进行排序与剪枝。在多个医疗推理基准上的实验表明,PRA持续优于强基线,在使用Qwen3-4B模型时于MedQA上达到81.9%准确率,创下4B规模新纪录。更重要的是,PRA可泛化至未见过的冻结策略模型,参数范围从0.5B到8B,准确率最高提升25.7%,且无需更新策略模型。更广泛而言,PRA提出一种范式:将冻结推理器与领域特定奖励模块解耦,使复杂领域可灵活更换主干模型而无需重训练。
原文摘要 · Abstract (English)
Reasoning in knowledge-intensive domains remains challenging as intermediate steps are often not locally verifiable: unlike math or code, evaluating step correctness may require synthesizing clues across large external knowledge sources. As a result, subtle errors can propagate through reasoning traces, potentially never to be detected. Prior work has proposed process reward models (PRMs), including retrieval-augmented variants, but these methods operate post hoc, scoring completed trajectories, which prevents their integration into dynamic inference procedures. Here, we introduce Process Reward Agents (PRA), an inference-time method for providing domain-grounded, online, step-wise rewards to a frozen policy. In contrast to prior retrieval-augmented PRMs, PRA enables search-based decoding to rank and prune candidate trajectories at every generation step. Experiments on multiple medical reasoning benchmarks demonstrate that PRA consistently outperforms strong baselines, achieving 81.9% accuracy on MedQA with Qwen3-4B, a new state of the art at the 4B scale. Importantly, PRA generalizes to unseen frozen policy models ranging from 0.5B to 8B parameters, improving their accuracy by up to 25.7% without any policy model updates. More broadly, PRA suggests a paradigm in which frozen reasoners are decoupled from domain-specific reward modules, allowing the deployment of new backbones in complex domains without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。