通过每轮推理的潜在信息量设计奖励，让搜索增强大模型训练更稳定高效。
TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs
- 基于教师模型评估每轮推理+工具调用对答案正确性的提升潜力，生成细粒度奖励
- 在7个QA基准上，相比PPO提升11.8%精确匹配率和13.6%F1值
- 适合需要多轮推理与工具调用的开放域问答任务研究者
搜索增强的大语言模型（LLMs）在开放域问答任务中已取得优异表现，但强化学习训练仍面临挑战。由于奖励稀疏且难以在推理与工具调用间合理分配责任，优化过程常不稳定。为此，我们提出逐轮信息潜力奖励塑造（TIPS），一种简单框架：依据教师模型下正确答案概率的提升，为每个推理+工具调用阶段分配密集的逐轮奖励。通过潜在函数奖励塑造，TIPS提供细粒度且策略无关的指导，克服了仅基于结果优化的局限性。在7个问答基准上评估显示，TIPS持续优于GRPO/PPO基线，并显著提升训练稳定性。例如，在Qwen-2.5 7B Instruct模型上，平均精确匹配率提升11.8%，F1值提升13.6%。结果表明，逐轮信息潜力奖励塑造为多轮推理中的稀疏奖励信用分配提供了有效且通用的解决方案。
原文摘要 · Abstract (English)
Search-augmented large language models (LLMs) trained with reinforcement learning (RL) have achieved strong results on open-domain question answering (QA), but training still remains a significant challenge. The optimization is often unstable due to sparse rewards and difficult credit assignments across reasoning and tool calls. To address this, we introduce Turn-Level Information Potential Reward Shaping (TIPS), a simple framework that assigns dense, turn-level rewards to each reasoning + tool-call segment based on the increased likelihood of the correct answer under a teacher model. By leveraging the potential-based reward shaping, TIPS offers fine-grained and policy-invariant guidance that overcomes the limitations of outcome-only optimization. Evaluated on seven QA benchmarks, TIPS consistently outperforms GRPO/PPO baselines and substantially improves training stability. For instance, with a Qwen-2.5 7B Instruct model, TIPS improves the average Exact Match score by 11.8% and F1 by 13.6% relative to PPO. Our results demonstrate that turn-level information-potential reward shaping provides an effective and general solution to sparse-reward credit assignment for multi-turn LLM reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。