用覆盖度奖励引导大模型拆解复杂问题，提升搜索代理的推理能力。
PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent
- 通过教师模型分解复杂查询，生成关键搜索项作为指导信号。
- 在R1框架下训练，任务完成率显著提升，覆盖度奖励有效改善策略。
- 适合需要精准拆解问题的智能搜索与决策类应用开发者。
现有搜索增强型大模型代理多采用强化学习提升推理能力，但主要依赖结果层面的奖励，对搜索行为监督不足，且忽视代理正确分解复杂查询的能力。为此，我们提出PROGRESS，利用教师引导的覆盖度奖励，显式优化策略模型的查询分解过程。训练时，固定教师模型将复杂查询分解为必要搜索项，并用于指导策略模型的搜索行为。该方法集成于R1风格训练框架，无需密集的过程级监督即可提供轻量级指导。实验表明，覆盖度引导的强化学习显著提升了整体任务表现，凸显了在智能体大模型中显式监督搜索策略的重要性。
原文摘要 · Abstract (English)
Existing search-augmented LLM agents are trained using Reinforcement Learning to boost its reasoning capabilities. However, these approaches primarily rely on outcome-level rewards, which provide little supervision over search behavior and overlook agent's ability to decompose complex queries properly. To mitigate this issue, we propose PROGRESS which utilizes teacher-guided coverage reward to explicitly shape decomposed query generation of the policy model. During training, frozen teacher models are used to decompose complex queries into essential search queries. These essential search queries are utilized to guide the search behavior of the policy model. Integrated into an R1-style training framework, our approach provides lightweight guidance over query decomposition decisions without dense process-level supervision. Experiments show that coverage-guided RL improves overall task performance, highlighting the importance of explicitly supervising search strategies in agentic LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。