提出DSPO算法，让大模型更稳定高效地自主搜索与推理。
DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
- 通过序列级优化和动态样本筛选提升强化学习稳定性
- 7B模型在多跳问答上比之前方法提升34.1%，超越14B模型近9%
- 无需监督数据，可纯强化学习训练，适合复杂交互任务
将主动检索外部知识的能力赋予大语言模型，对完成复杂现实任务至关重要。现有方法或依赖提示激发模型的内在代理能力，或在应用于复杂交互任务时面临性能瓶颈与崩溃问题，难以发挥其真正的代理潜力。为此，我们提出动态序列级策略优化（DSPO），一种改进的强化学习算法，通过序列级优化与动态样本过滤实现鲁棒的代理训练。我们仅用强化学习训练模型，使其能自然地交织多轮搜索与推理，无需监督演示数据。在多个问答基准测试中，我们的7B模型相比先前工作提升34.1%，在复杂多跳问答（如HotpotQA）中甚至接近9%相对优于此前工作的14B模型，且训练过程保持极佳稳定性。
原文摘要 · Abstract (English)
Enhancing LLMs with the ability to actively search external knowledge is crucial for complex and real-world tasks. Current approaches either rely on prompting to elicit the model's innate agent capabilities, or suffer from performance ceilings and collapse when applying RL to complex interactive tasks, leaving their true agentic potential untapped. To address this, we introduce \textbf{D}ynamic-filter \textbf{S}equence-level \textbf{P}olicy \textbf{O}ptimization (DSPO), an improved RL algorithm designed for robust agent training through sequence-level optimization and dynamic sample filtering. We train our model purely through RL to interleave multi-turn search and reasoning, obviating the need for supervised demonstration data. Across multiple QA benchmarks, our 7B model improves over a comparable previous work by \textbf{34.1\%}, and even outperforms the 14B model from previous work in complex multihop QA such as HotpotQA by nearly \textbf{9\% relative}, maintaining exceptional training stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。