通过解耦提示、奖励与策略优化，提升深度研究智能体的检索决策能力。
How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1
- 采用快思模板替代慢思模板，提升稳定性和性能。
- 引入动作级惩罚缓解答案回避问题，使基于F1的奖励超越EM指标。
- 使用REINFORCE策略优化，在更少搜索次数下实现更好效果。
深度研究智能体通过多轮检索与面向决策的生成来处理知识密集型任务。尽管强化学习（RL）已被证明可提升此类系统性能，但其作用仍不明确。本文从提示模板、奖励函数和策略优化三个独立维度进行系统性研究：1）快思模板比先前工作中的慢思模板更具稳定性且表现更优；2）基于F1的奖励因答案回避导致训练崩溃而表现不佳，通过引入动作级惩罚可有效缓解，最终超越EM；3）REINFORCE在减少搜索动作的同时优于PPO，而GRPO在策略优化方法中稳定性最差。基于上述发现，我们提出Search-R1++，作为强基线模型，将Search-R1在Qwen2.5-7B上的性能从0.403提升至0.442，在Qwen2.5-3B上从0.289提升至0.331。期望这些发现能推动深度研究系统中更系统、可靠的强化学习训练策略发展。
原文摘要 · Abstract (English)
Deep Research agents tackle knowledge-intensive tasks through multi-round retrieval and decision-oriented generation. While reinforcement learning (RL) has been shown to improve performance in this paradigm, its contributions remain underexplored. To fully understand the role of RL, we conduct a systematic study along three decoupled dimensions: prompt template, reward function, and policy optimization. Our study reveals that: 1) the Fast Thinking template yields greater stability and better performance than the Slow Thinking template used in prior work; 2) the F1-based reward underperforms the EM due to training collapse driven by answer avoidance; this can be mitigated by incorporating action-level penalties, ultimately surpassing EM; 3) REINFORCE outperforms PPO while requiring fewer search actions, whereas GRPO shows the poorest stability among policy optimization methods. Building on these insights, we then introduce Search-R1++, a strong baseline that improves the performance of Search-R1 from 0.403 to 0.442 (Qwen2.5-7B) and 0.289 to 0.331 (Qwen2.5-3B). We hope that our findings can pave the way for more principled and reliable RL training strategies in Deep Research systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。