arXiv:2602.23440cs.CLcs.IR2026-02

通过截断采样与细粒度奖励,提升检索增强推理的决策精度。

Truncated Step-Level Sampling with Process Rewards for Retrieval-Augmented Reasoning

  • 截断采样仅让单步变化,降低策略梯度方差。
  • 使用三元评分对推理、查询、答案分别打分,监督更精细。
  • 在多跳问答上效果显著,小模型提升超30%。

强化学习已成为训练大语言模型在推理中穿插调用搜索引擎的有效范式。然而,现有方法存在根本性的信用分配难题:如Search-R1对整个多步轨迹给予单一结果奖励,无法判断哪一步决策导致成败。类似StepSearch的步骤级奖励方法虽引入步骤级监督,但仍独立采样完整轨迹,使任一步的优劣估计受其他步骤随机性干扰。本文提出SLATE(截断探索的步骤级优势估计),通过两项互补设计解决上述问题。首先,截断步骤级采样从共享前缀生成k个延续路径,将所有差异隔离于单一决策点,理论上可将优势估计方差降低至全轨迹采样的1/T,首次为检索增强推理中的步骤级强化学习提供正式方差保障。其次,采用由LLM裁判生成的密集分解过程奖励,以三元尺度分别评估推理质量、查询质量与答案正确性,提供比二值结果信号或启发式得分更丰富的监督。在七个QA基准测试中,SLATE持续优于稀疏奖励与过程奖励基线,在7B模型上相对Search-R1提升7.0%,在3B模型上提升30.7%。性能增益在挑战性的多跳任务中最为明显,消融实验确认截断采样与密集奖励具有互补优势。

原文摘要 · Abstract (English)

Reinforcement learning has emerged as an effective paradigm for training large language models to interleave reasoning with search engine calls. However, existing approaches face a fundamental credit assignment problem: methods like Search-R1 assign a single outcome reward to the entire multi-step trajectory, providing no signal about which reasoning or retrieval decisions were responsible for success or failure. Process-reward methods such as StepSearch introduce step-level supervision but still sample complete trajectories independently, so advantage estimates at any given step are contaminated by the randomness of all other steps. We propose SLATE (Step-Level Advantage estimation for Truncated Exploration), which addresses both problems through two complementary ideas. First, truncated step-level sampling generates k continuations from a shared prefix, isolating all variation to a single decision point. We prove this reduces the variance of advantage estimates by up to a factor of T compared to full-trajectory sampling for T-step trajectories, the first formal variance guarantee for step-level RL in retrieval-augmented reasoning. Second, dense, decomposed process rewards separately evaluate reasoning quality, query quality, and answer correctness on a ternary scale via an LLM judge, providing richer supervision than binary outcome signals or heuristic step-level scores. Experiments on seven QA benchmarks show that SLATE consistently outperforms both sparse-reward and process-reward baselines, achieving a 7.0% relative improvement over Search-R1 on the 7B model and 30.7% on the 3B model. Gains are largest on challenging multi-hop tasks, and ablations confirm that truncated sampling and dense rewards provide complementary benefits.

强化学习检索增强推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。