解决多轮搜索中奖励稀疏导致的思维同质化问题
TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization
- 引入首次出现隐式奖励机制，保留过程级信号
- 在Qwen2.5-3B和7B上分别提升24%和13.6%
- 无需外部标注或奖励模型，适合复杂任务优化
多轮工具集成推理使大语言模型通过迭代检索解决复杂任务。然而，当前基于强化学习的搜索增强推理框架主要依赖稀疏的结果级奖励，导致“双重同质化困境”：（1）过程同质化，生成中的思考、推理和工具使用被忽略；（2）组内同质化，粗粒度结果奖励在采样时导致组内优势估计效率低下，如GRPO方法。为此，我们提出分阶段感知策略优化（TSPO），引入首次出现隐式奖励（FOLR）机制，将部分奖励分配给真实答案首次出现的步骤，从而在不依赖外部奖励模型或标注的情况下，保留过程级信号并提升组内奖励方差。大量实验表明，TSPO显著优于现有基线，在Qwen2.5-3B和7B模型上分别实现平均性能提升24%和13.6%。代码已公开于https://github.com/Flipped-May/TSPO。
原文摘要 · Abstract (English)
Multi-turn tool-integrated reasoning enables Large Language Models (LLMs) to solve complex tasks through iterative information retrieval. However, current reinforcement learning (RL) frameworks for search-augmented reasoning predominantly rely on sparse outcome-level rewards, leading to a "Double Homogenization Dilemma." This manifests as (1) Process homogenization, where the thinking, reasoning, and tooling involved in generation are ignored. (2) Intra-group homogenization, coarse-grained outcome rewards often lead to inefficiencies in intra-group advantage estimation with methods like Group Relative Policy Optimization (GRPO) during sampling. To address this, we propose Turn-level Stage-aware Policy Optimization (TSPO). TSPO introduces the First-Occurrence Latent Reward (FOLR) mechanism, allocating partial rewards to the step where the ground-truth answer first appears, thereby preserving process-level signals and increasing reward variance within groups without requiring external reward models or any annotations. Extensive experiments demonstrate that TSPO significantly outperforms state-of-the-art baselines, achieving average performance gains of 24% and 13.6% on Qwen2.5-3B and 7B models, respectively. Code is available at https://github.com/Flipped-May/TSPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。