用分步奖励训练大模型搜索，提升复杂问答准确率
StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization
- 采用分步近端策略优化，每一步提供细化奖励
- 3B/7B模型在多跳问答上分别提升11.2%和4.2%
- 适合需要深度搜索推理的问答系统开发者
高效多跳推理需要大语言模型代理能迭代获取高价值外部知识。现有研究通过强化学习训练模型进行基于搜索的文档检索，在问答性能上取得显著提升，但在复杂多跳问答任务中表现不佳，主要因仅依赖全局奖励信号导致奖励稀疏。为此，我们提出StepSearch框架，利用分步近端策略优化方法训练搜索型大模型。该框架引入更丰富细致的中间搜索奖励，并基于信息增益与冗余惩罚进行逐标记过程监督，以更好指导每一步搜索。我们通过一套数据管道方法，在开源数据集基础上构建了包含子问题级搜索轨迹的细粒度问答数据集。在标准多跳问答基准测试中，该方法显著优于全局奖励基线，在仅使用19,000条训练数据的情况下，使3B和7B模型分别实现11.2%和4.2%的绝对性能提升，证明了细粒度、分步监督在优化深度搜索大模型中的有效性。代码将发布于https://github.com/Zillwang/StepSearch。
原文摘要 · Abstract (English)
Efficient multi-hop reasoning requires Large Language Models (LLMs) based agents to acquire high-value external knowledge iteratively. Previous work has explored reinforcement learning (RL) to train LLMs to perform search-based document retrieval, achieving notable improvements in QA performance, but underperform on complex, multi-hop QA resulting from the sparse rewards from global signal only. To address this gap in existing research, we introduce StepSearch, a framework for search LLMs that trained with step-wise proximal policy optimization method. It consists of richer and more detailed intermediate search rewards and token-level process supervision based on information gain and redundancy penalties to better guide each search step. We constructed a fine-grained question-answering dataset containing sub-question-level search trajectories based on open source datasets through a set of data pipeline method. On standard multi-hop QA benchmarks, it significantly outperforms global-reward baselines, achieving 11.2% and 4.2% absolute improvements for 3B and 7B models over various search with RL baselines using only 19k training data, demonstrating the effectiveness of fine-grained, stepwise supervision in optimizing deep search LLMs. Our code will be released on https://github.com/Zillwang/StepSearch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。