通过过程与结果奖励结合，提升大模型检索生成中的思维推理能力。
LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization
- 设计过程奖励模块，让模型关注每一步思考与搜索的正确性。
- 在多个RAG基准上实现更强泛化与推理效率，效果优于纯结果奖励。
- 适合需要精准推理的智能问答、知识检索场景，可推广至其他强化学习任务。
大型语言模型（LLMs）在推理方面展现出强大能力，如OpenAI-o1和DeepSeek-R1等推理模型的出现。近期研究通过基于结果的强化学习（RL）方法，将推理能力融入检索增强生成（RAG）领域，但通常忽视中间思考与搜索步骤的正确性。为解决此问题，我们设计了一个过程级奖励模块，无需额外标注即可缓解结果级监督对中间推理步骤的忽视。在此基础上，提出一种新框架LeTS，将步骤级过程奖励与结果级奖励相结合，应用于现有RAG的强化学习方法中。大量实验表明，LeTS在多个RAG基准上具有优异的泛化能力和推理效率。此外，这些结果揭示了过程与结果级奖励融合在其他强化学习场景下提升大模型推理能力的潜力。代码即将开源。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated impressive capabilities in reasoning with the emergence of reasoning models like OpenAI-o1 and DeepSeek-R1. Recent research focuses on integrating reasoning capabilities into the realm of retrieval-augmented generation (RAG) via outcome-supervised reinforcement learning (RL) approaches, while the correctness of intermediate think-and-search steps is usually neglected. To address this issue, we design a process-level reward module to mitigate the unawareness of intermediate reasoning steps in outcome-level supervision without additional annotation. Grounded on this, we propose Learning to Think-and-Search (LeTS), a novel framework that hybridizes stepwise process reward and outcome-based reward to current RL methods for RAG. Extensive experiments demonstrate the generalization and inference efficiency of LeTS across various RAG benchmarks. In addition, these results reveal the potential of process- and outcome-level reward hybridization in boosting LLMs' reasoning ability via RL under other scenarios. The code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。