arXiv:2608.22479cs.CLcs.LG2026-08中稿 · EMNLP

用图轨迹增强强化学习,让大模型更准地找到多跳问答的证据链。

GTA-RAG: Graph-Trajectory-Augmented Reinforcement Learning for Multi-Turn Retrieval-Augmented Reasoning

论文配图:GTA-RAG: Graph-Trajectory-Augmented Reinforcement Learning for Multi-Turn Retrieval-Augmented Reasoning
图 1 · 摘自论文原文
  • 构建文档-实体图,生成可执行的多跳推理路径作为训练信号。
  • 在三个多跳和两个简单问答数据集上超越现有强化学习基线。
  • 适合研究智能体推理、知识检索与强化学习融合的学者。

检索增强生成(RAG)使大语言模型能访问外部知识以回答知识密集型问题。对于复杂的多跳问题,多轮检索增强推理将RAG扩展为迭代过程,反复搜索并整合文档中的证据。然而,现有的基于强化学习(RL)的代理式RAG通常仅以最终答案奖励进行优化,这导致监督信号稀疏,并忽略了模型是否真正获取了所需的证据链。本文提出 extsc{GTA-RAG},一种图-轨迹增强的强化学习框架,用于多轮检索增强推理。从实体-文档图中采样连通文档路径,合成多跳问答轨迹,并通过部署的检索器验证其可执行性,从而获得轨迹级监督信号。随后,采用分组相对策略优化(GRPO)和轨迹引导奖励,同时鼓励准确回答和目标证据文档的获取,最后在自然问答实例上进行答案奖励训练。在三个多跳和两个简单问答基准上的实验表明,该方法在使用Qwen2.5-3B和Qwen2.5-7B两种模型时均持续优于基于RL的RAG基线,且显著提升证据链覆盖率。

原文摘要 · Abstract (English)

Retrieval-augmented generation (RAG) enables LLMs to access external knowledge for answering knowledge-intensive questions. For complex multi-hop questions, multi-turn retrieval-augmented reasoning extends RAG into an iterative process that repeatedly searches for and integrates evidence across documents. However, existing reinforcement-learning (RL) approaches for agentic RAG are typically optimized with final-answer rewards, which provide sparse supervision and overlook whether the model actually retrieves the required evidence chain. We present \textsc{GTA-RAG}, a graph-trajectory-augmented RL framework for multi-turn retrieval-augmented reasoning. From an entity--document graph, we sample connected document paths, synthesize multi-hop QA trajectories, and validate them with the deployed retriever to obtain executable trajectory-level supervision. We then optimize the retrieval policy with Group Relative Policy Optimization (GRPO) and a trajectory-guided reward that encourages both accurate answers and acquisition of target evidence documents, followed by answer-reward training on natural QA instances. Experiments on three multi-hop and two simple QA benchmarks show that \method{} consistently outperforms RL-based RAG baselines with both Qwen2.5-3B and Qwen2.5-7B backbones, while substantially improving evidence-chain coverage. Our code is available at https://github.com/cjcj46262/GTA-RAG.

多跳推理强化学习检索增强知识图谱

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。