arXiv:2601.05465cs.AI2026-01被引 1

用强化学习分两阶段优化多智能体问答,解决复杂问题检索失败和训练不稳问题。

PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answering

  • 分阶段强化学习,智能体协同规划、检索、验证与求解。
  • 在10个基准上达到顶尖性能,推理准确率显著提升。
  • 适合需要高可靠性的开放域多跳问答系统部署。

在大规模语料上回答现实世界的开放域多跳问题,是检索增强生成(RAG)系统的关键挑战。现有研究采用强化学习端到端优化检索-增强推理过程,直接提升其解决复杂查询的能力。然而,实际部署面临两大障碍:1)检索崩溃:在大规模语料中迭代检索时,缺乏基于推理的规划,无法定位包含桥接答案的中间证据,导致下游推理失效;2)学习不稳定性:端到端轨迹训练存在信用分配弱、模块错误定位差的问题,导致过拟合于特定基准的启发式规则,影响迁移性和稳定性。为此,我们提出PRISMA,一种解耦的强化学习引导框架,采用计划-检索-检查-求解-记忆架构。其核心在于推理引导协作:检查器提供基于推理的反馈,优化规划器的分解与细粒度检索,并确保求解器的证据依赖推理。通过两阶段群体相对策略优化(GRPO)分别优化各智能体能力:第一阶段将规划器与求解器训练为专业专家;第二阶段采用观测感知残差策略优化(OARPO),增强检查器验证上下文和触发针对性恢复的能力。实验表明,PRISMA在10个基准上达到最先进水平,可高效部署于真实场景。

原文摘要 · Abstract (English)

Answering real-world open-domain multi-hop questions over massive corpora is a critical challenge in Retrieval-Augmented Generation (RAG) systems. Recent research employs reinforcement learning (RL) to end-to-end optimize the retrieval-augmented reasoning process, directly enhancing its capacity to resolve complex queries. However, reliable deployment is hindered by two obstacles. 1) Retrieval Collapse: iterative retrieval over large corpora fails to locate intermediate evidence containing bridge answers without reasoning-guided planning, causing downstream reasoning to collapse. 2) Learning Instability: end-to-end trajectory training suffers from weak credit assignment across reasoning chains and poor error localization across modules, causing overfitting to benchmark-specific heuristics that limit transferability and stability. To address these problems, we propose PRISMA, a decoupled RL-guided framework featuring a Plan-Retrieve-Inspect-Solve-Memoize architecture. PRISMA's strength lies in reasoning-guided collaboration: the Inspector provides reasoning-based feedback to refine the Planner's decomposition and fine-grained retrieval, while enforcing evidence-grounded reasoning in the Solver. We optimize individual agent capabilities via Two-Stage Group Relative Policy Optimization (GRPO). Stage I calibrates the Planner and Solver as specialized experts in planning and reasoning, while Stage II utilizes Observation-Aware Residual Policy Optimization (OARPO) to enhance the Inspector's ability to verify context and trigger targeted recovery. Experiments show that PRISMA achieves state-of-the-art performance on ten benchmarks and can be deployed efficiently in real-world scenarios.

多跳问答强化学习多智能体RAG

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。