用强化学习让大模型自主查图、推理,解决复杂问题更准更快。
GraphRAG-R1: Graph Retrieval-Augmented Generation with Process-Constrained Reinforcement Learning
- 通过过程约束的强化学习,让模型自主规划检索与推理路径。
- 在多个数据集上显著提升多跳推理准确率,尤其在域外数据表现更好。
- 支持灵活接入现有检索方法,适合需要深度推理的应用场景。
图检索增强生成(GraphRAG)通过图结构表示知识,有效提升大模型的推理能力。然而,现有方法在处理需多跳推理的复杂问题时仍受限,因其查询与检索阶段依赖预设启发式规则,未充分挖掘大模型的推理潜力。为此,我们提出GraphRAG-R1,一种基于过程约束的结果导向强化学习(RL)训练的大模型框架,以增强多跳推理能力。该方法可自动分解复杂问题,调用检索工具获取必要信息并进行有效推理。具体地,采用改进的分组相对策略优化(GRPO),支持带思考的采样能力;设计两种过程约束奖励函数:渐进式检索衰减(PRA)奖励以避免浅层检索,成本感知F1(CAF)奖励以平衡性能与计算开销;引入分阶段训练策略,涵盖冷启动及两类奖励对应阶段;最后采用混合图-文本检索机制提升推理能力。大量实验表明,GraphRAG-R1在领域内与领域外数据集上均优于现有最先进GraphRAG方法。此外,该框架可灵活集成多种现有检索方法,持续带来性能提升。
原文摘要 · Abstract (English)
Graph Retrieval-Augmented Generation (GraphRAG) has shown great effectiveness in enhancing the reasoning abilities of LLMs by leveraging graph structures for knowledge representation and modeling complex real-world relationships. However, existing GraphRAG methods still face significant bottlenecks when handling complex problems that require multi-hop reasoning, as their query and retrieval phases are largely based on pre-defined heuristics and do not fully utilize the reasoning potentials of LLMs. To address this problem, we propose GraphRAG-R1, an adaptive GraphRAG framework by training LLMs with process-constrained outcome-based reinforcement learning (RL) to enhance the multi-hop reasoning ability. Our method can decompose complex problems, autonomously invoke retrieval tools to acquire necessary information, and perform effective reasoning. Specifically, we utilize a modified version of Group Relative Policy Optimization (GRPO) that supports rollout-with-thinking capability. Next, we design two process-constrained reward functions. To handle the shallow retrieval problem, we design a Progressive Retrieval Attenuation (PRA) reward to encourage essential retrievals. Then, to handle the over-thinking problem, we design Cost-Aware F1 (CAF) reward to balance the model performance with computational costs. We further design a phase-dependent training strategy, containing three training stages corresponding to cold start and these two rewards. Lastly, our method adopts a hybrid graph-textual retrieval to improve the reasoning capacity. Extensive experimental results demonstrate that GraphRAG-R1 boosts LLM capabilities in solving complex reasoning problems compared to state-of-the-art GraphRAG methods on both in-domain and out-of-domain datasets. Furthermore, our framework can be flexibly integrated with various existing retrieval methods, consistently delivering performance improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。