用合成数据训练模型分步搜索，提升复杂问题回答能力。
$S^3$-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data
- 构建合成数据流水线，生成多跳、中等难度问题
- 设计中间搜索质量与最终答案正确性双重奖励机制
- 在跨领域测试中提升10%泛化性能，适合复杂问答场景
强化学习后训练使模型具备搜索工具的代理能力，但受限于稀疏的基于结果的奖励和缺乏不同难度的问题训练数据，导致模型难以进行深度搜索以收集证据。为此，我们提出S^3-R1（合成数据与稳定搜索R1），结合数据驱动方法与更密集的学习信号。首先，开发了一套程序化生成与筛选流水线，从现有文档中衍生出多样化的多跳问题，并通过检索验证步骤专门筛选出中等难度的问题。随后，将扩展后的训练集与评估中间搜索质量及最终答案正确性的奖励结构结合，直接缓解稀疏奖励下的信用分配难题。实验表明，S^3-R1优于现有基线，学习到更有效的搜索与整合策略，在跨领域数据集上实现高达10%的鲁棒泛化提升。
原文摘要 · Abstract (English)
Reinforcement learning (RL) post-training has enabled newer capabilities in models, such as agentic tool-use for search. However, these models struggle primarily due to limitations with sparse outcome-based rewards and a lack of training data that encapsulates questions of differing hardness, which results in models not performing deeper searches with tools to collect evidence for question-answering. To address these limitations, we introduce S^3-R1 (Synthetic data and stabilized Search R1), a framework that couples a data-centric approach with denser learning signals. We first develop a synthetic generation and curation pipeline that programmatically derives diverse, multi-hop questions from existing documents. This pipeline incorporates a retrieval-based verification step to specifically isolate questions of intermediate difficulty. We then pair this expanded training set with a reward structure that evaluates both intermediate search quality and the correctness of the final answer. This setup directly mitigates the credit assignment problems inherent to sparse rewards. Our evaluations show that S^3-R1 outperforms existing baselines by learning more effective search and synthesis strategies, yielding up to a 10% improvement in robust generalization on out-of-domain datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。