用强化学习优化多跳检索,让大模型更会分步推理找答案。
OPERA: A Reinforcement Learning--Enhanced Orchestrated Planner-Executor Architecture for Reasoning-Oriented Multi-Hop Retrieval
- 分步规划+执行架构,自动拆解复杂问题并精准检索。
- 在多个多跳问答数据集上超越现有方法,准确率提升显著。
- 适合需要深度推理的智能问答系统开发者使用。
大型语言模型(LLMs)与密集检索器的进步推动了检索增强生成(RAG)的发展。然而,现有方法在复杂的推理导向多跳检索任务中仍面临三大挑战:1)无效的推理导向规划:基于规则的分解器对非模板化问题表现不佳;2)次优的推理驱动检索:查询改写能力有限,导致迭代检索循环难以找到正确文档;3)不足的推理引导过滤:缺乏细粒度推理,无法从噪声结果中有效筛选关键信息。根本原因在于当前RAG架构中检索与推理耦合不足。本文提出协同规划-执行推理架构(OPERA),其目标规划模块(GPM)将问题分解为子目标,由具备精确推理与高效检索能力的推理-执行模块(REM)执行。为训练OPERA,我们提出多智能体渐进组相对策略优化(MAPGRPO),一种GRPO新变体。在复杂多跳基准测试上的实验表明,OPERA性能显著优于基线,验证了方法与架构设计的有效性。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) and dense retrievers have driven significant progress in retrieval-augmented generation (RAG). However, existing approaches face significant challenges in complex reasoning-oriented multi-hop retrieval tasks: 1) Ineffective reasoning-oriented planning: Prior methods struggle to generate robust multi-step plans for complex queries, as rule-based decomposers perform poorly on out-of-template questions. 2) Suboptimal reasoning-driven retrieval: Related methods employ limited query reformulation, leading to iterative retrieval loops that often fail to locate golden documents. 3) Insufficient reasoning-guided filtering: Prevailing methods lack the fine-grained reasoning to effectively filter salient information from noisy results, hindering utilization of retrieved knowledge. Fundamentally, these limitations all stem from the weak coupling between retrieval and reasoning in current RAG architectures. We introduce the Orchestrated Planner-Executor Reasoning Architecture (OPERA), a novel reasoning-driven retrieval framework. OPERA's Goal Planning Module (GPM) decomposes questions into sub-goals, which are executed by a Reason-Execute Module (REM) with specialized components for precise reasoning and effective retrieval. To train OPERA, we propose Multi-Agents Progressive Group Relative Policy Optimization (MAPGRPO), a novel variant of GRPO. Experiments on complex multi-hop benchmarks show OPERA's superior performance, validating both the MAPGRPO method and OPERA's design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。