arXiv:2603.13853cs.CLcs.AI2026-03

通过分层奖励分离规划与执行,提升复杂问答的检索生成效果。

APEX-Searcher: Refining Credit Assignment with Subgoaling for Agentic Retrieval-Augmented Generation

  • 用计划级奖励优化规划,执行级监督微调学习检索
  • 多跳任务上相比基线提升12.3%准确率
  • 适合需要精细推理和可靠检索的智能助手场景

检索增强生成(RAG)将大语言模型与外部知识连接,但单轮检索难以应对复杂的多跳问题。现有方法通常通过端到端强化学习整合多轮迭代检索与推理,虽提升解题性能,但仍面临任务推理和模型训练挑战,尤其在检索路径模糊和强化学习稀疏奖励下,易导致检索结果不准、性能下降。我们归因于层级信用纠缠:最终奖励同时更新规划与执行,使模型无法区分计划错误与检索错误。为此提出APEX-Searcher,采用精炼信用分配范式:规划通过强化学习以计划级奖励优化,执行通过监督微调学习。大量实验表明,在多个基准上,多跳RAG与任务规划均取得持续提升。

原文摘要 · Abstract (English)

Retrieval-augmented generation (RAG) connects large language models (LLMs) to external knowledge, but single-round retrieval is often insufficient for complex multi-hop questions. To enhance search capabilities for complex tasks, most existing works integrate multi-round iterative retrieval with reasoning processes via end-to-end training. While these approaches improve problem-solving performance, they still face challenges in task reasoning and model training, especially ambiguous retrieval execution paths and sparse rewards in end-to-end reinforcement learning (RL), which can lead to inaccurate retrieval results and lower performance. We attribute these failures to hierarchical credit entanglement: a single final reward updates planning and execution together, so the model cannot clearly separate plan errors from retrieval errors. We propose APEX-Searcher, which uses a Refining Credit Assignment paradigm: planning is optimized by RL with a plan-level reward, while execution is learned by SFT. Extensive experiments show consistent gains in both multi-hop RAG and task planning across benchmarks.

检索增强多跳问答强化学习自主代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。