用验证器反馈指导大模型搜索，提升复杂问题求解准确率
CAPF: Guiding Search-Agent Rollouts with Credit-Attenuated Privileged Feedback
- 训练时引入验证器的纠错反馈，动态修正失败推理路径
- 在7个开放域问答数据集上，准确率从44.7%提升至48.5%
- 适合需要高精度推理的复杂任务场景，如智能问答与自动规划
近期基于大语言模型的搜索代理采用可验证奖励的强化学习（RLVR）来学习从结果奖励中进行搜索增强型推理。在难题上，这些代理很少采样到端到端成功的推理路径，导致仅依赖结果奖励的RLVR缺乏正向奖励轨迹。我们认为，提升此类问题的学习效果需要训练期间的额外引导，而RLVR已包含可提供引导的验证器侧信息。该信息能识别代理提交答案中的错误或遗漏，并指导推理过程中的修正。我们提出一种训练期机制——信用衰减特权反馈（CAPF），通过训练期间的特权反馈调用，使验证器侧信息可用。CAPF可将零奖励尝试转化为正向奖励修复路径，并对反馈调用及早期动作进行信用衰减，以适应无此调用的部署环境。实证研究显示，CAPF将Qwen3-4B在七个开放域问答基准上的平均精确匹配得分从44.7%提升至48.5%。
原文摘要 · Abstract (English)
Recent LLM search agents use reinforcement learning with verifiable rewards (RLVR) to learn search-augmented reasoning from outcome rewards. On hard problems, these agents rarely sample end-to-end successful rollouts, leaving outcome-only RLVR with few positive-reward trajectories. We argue that improving learning on such problems requires additional guidance during training, and RLVR already contains verifier-side information that can provide it. This information can identify errors or omissions in the agent's submitted answer and guide revision within the rollout. We propose a training-time mechanism called \textbf{Credit-Attenuated Privileged Feedback} (CAPF), which makes this verifier-side information available through a Privileged Feedback call during training. CAPF lets the policy revise zero-reward attempts into positive-reward repair trajectories and attenuates credit for the feedback call and earlier actions to accommodate deployment without this call. Empirical research demonstrates that CAPF improves Qwen3-4B's average exact-match score from 44.7% under outcome-only RLVR to 48.5% on seven open-domain QA benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。