用课程引导强化学习优化多跳检索生成，减少冗余查询
Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation
- 设计渐进式奖励机制，分阶段引导查询重写
- 在四个数据集上提升准确率最高4.6点，检索深度降低15%
- 适合构建高效可靠的多跳问答系统
检索增强生成（RAG）将大语言模型与外部最新证据结合，但现有多跳RAG流程仍存在冗余子查询、探索不足或搜索链过长的问题。本文提出EVO-RAG，一种课程引导的强化学习框架，使查询重写代理从初期广域探索逐步过渡到后期精炼优化。该框架采用七因子步级奖励向量（涵盖相关性、冗余度、效率和答案正确性），并配合随时间动态调整的调度器，重新加权各信号。代理通过直接偏好优化训练多头奖励模型，学会何时搜索、回溯、回答或拒绝。在四个多跳问答基准（HotpotQA、2WikiMultiHopQA、MuSiQue、Bamboogle）上，EVO-RAG相比强基线提升精确匹配最高4.6点，平均检索深度减少15%。消融实验验证了课程阶段与动态奖励调度的互补作用。EVO-RAG为构建可靠、低成本的多跳RAG系统提供了通用方案。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) grounds large language models (LLMs) in up-to-date external evidence, yet existing multi-hop RAG pipelines still issue redundant subqueries, explore too shallowly, or wander through overly long search chains. We introduce EVO-RAG, a curriculum-guided reinforcement learning framework that evolves a query-rewriting agent from broad early-stage exploration to concise late-stage refinement. EVO-RAG couples a seven-factor, step-level reward vector (covering relevance, redundancy, efficiency, and answer correctness) with a time-varying scheduler that reweights these signals as the episode unfolds. The agent is trained with Direct Preference Optimization over a multi-head reward model, enabling it to learn when to search, backtrack, answer, or refuse. Across four multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle), EVO-RAG boosts Exact Match by up to 4.6 points over strong RAG baselines while trimming average retrieval depth by 15 %. Ablation studies confirm the complementary roles of curriculum staging and dynamic reward scheduling. EVO-RAG thus offers a general recipe for building reliable, cost-effective multi-hop RAG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。