通过路径奖励设计，让AI推理更稳定高效
Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training
- 以推理路径结构为评估核心，从失败样本中提取学习信号
- 在多个问答数据集上平均准确率提升7.7个百分点
- 适合需要多步推理的智能问答系统开发者
检索增强生成（RAG）通过引入外部知识提升大语言模型能力，但传统单轮检索难以应对复杂多步推理。代理式RAG通过让模型动态决定何时及检索什么内容来解决此问题，但现有基于强化学习的训练方法存在奖励稀疏、忽略中间信号、样本效率低等问题。本文提出Search-P1框架，包含两个关键组件：(1) 路径中心奖励，通过无序步骤覆盖和软评分机制评估推理轨迹结构质量，即使在失败样本中也能提取学习信号；(2) 双轨路径评分，结合离线生成的参考规划器，从自洽性和参考对齐双重角度评估路径。在多个QA基准上的实验表明，Search-P1相比Search-R1及其他强基线取得显著提升，平均准确率提高7.7点。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by incorporating external knowledge, yet traditional single-round retrieval struggles with complex multi-step reasoning. Agentic RAG addresses this by enabling LLMs to dynamically decide when and what to retrieve, but current RL-based training methods suffer from sparse outcome rewards that discard intermediate signals and low sample efficiency where failed samples contribute nothing. We propose Search-P1, a framework that introduces path-centric reward shaping for agentic RAG training, comprising two key components: (1) Path-Centric Reward, which evaluates the structural quality of reasoning trajectories through order-agnostic step coverage and soft scoring that extracts learning signals even from failed samples, and (2) Dual-Track Path Scoring with offline-generated reference planners that assesses paths from both self-consistency and reference-alignment perspectives. Experiments on multiple QA benchmarks demonstrate that Search-P1 achieves significant improvements over Search-R1 and other strong baselines, with an average accuracy gain of 7.7 points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。