让大模型先规划再检索,复杂问题问答更准更智能
Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs
- 先规划后检索,分步拆解问题并决定何时查知识图谱或网页
- 在不完整知识图谱下,准确率提升12.3%,多跳问答效果显著
- 适合需要跨源推理的复杂问答任务,如医疗、科研场景
知识图谱问答旨在通过结构化知识图谱回答自然语言问题。尽管大语言模型凭借强大的推理能力推动了KGQA发展,但现有方法仍难以充分利用知识图谱中的丰富信息和大模型的推理能力,尤其在复杂场景下表现不佳。它们常假设知识图谱覆盖完整,缺乏判断是否需要外部信息的机制,且推理过程局部短视,无法保持连贯的多步规划,导致即使存在相关知识也出现推理失败。我们提出Graph-RFT,一种两阶段强化学习微调框架,采用‘计划-检索-搜索’范式,使大模型在知识不完整条件下实现自主规划与自适应跨源检索调度。Graph-RFT引入基于链式思维的微调方法及定制化的计划-检索数据集,激活结构化推理并解决GRPO冷启动问题;设计一种新的计划-检索引导式强化学习流程,将显式规划与检索动作结合,采用多奖励机制,实现覆盖感知的检索调度。通过受笛卡尔启发的规划模块,将复杂问题分解为有序子问题,并用逻辑表达式指导工具调用,实现全局一致的多步推理。该推理检索过程通过融合结果与检索特定信号的多奖励机制进行优化,使模型学会何时以及如何有效结合知识图谱与网络检索。
原文摘要 · Abstract (English)
Knowledge Graph Question Answering aims to answer natural language questions by reasoning over structured knowledge graphs. While large language models have advanced KGQA through their strong reasoning capabilities, existing methods continue to struggle to fully exploit both the rich knowledge encoded in KGs and the reasoning capabilities of LLMs, particularly in complex scenarios. They often assume complete KG coverage and lack mechanisms to judge when external information is needed, and their reasoning remains locally myopic, failing to maintain coherent multi-step planning, leading to reasoning failures even when relevant knowledge exists. We propose Graph-RFT, a novel two-stage reinforcement fine-tuning KGQA framework with a 'plan-KGsearch-and-Websearch-during-think' paradigm, that enables LLMs to perform autonomous planning and adaptive retrieval scheduling across KG and web sources under incomplete knowledge conditions. Graph-RFT introduces a chain-of-thought fine-tuning method with a customized plan-retrieval dataset activates structured reasoning and resolves the GRPO cold-start problem. It then introduces a novel plan-retrieval guided reinforcement learning process integrates explicit planning and retrieval actions with a multi-reward design, enabling coverage-aware retrieval scheduling. It employs a Cartesian-inspired planning module to decompose complex questions into ordered subquestions, and logical expression to guide tool invocation for globally consistent multi-step reasoning. This reasoning retrieval process is optimized with a multi-reward combining outcome and retrieval specific signals, enabling the model to learn when and how to combine KG and web retrieval effectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。