用强化学习让小模型一次完成多跳知识推理,效果超大模型。
KG-Hopper: Empowering Compact Open LLMs with Knowledge Graph Reasoning via Reinforcement Learning
- 训练小模型在单轮内全局规划知识图谱路径,支持回溯和动态调整
- 70亿参数模型在8个基准上超越700亿参数的多步系统
- 适合资源有限但需精准推理的应用场景,开源可复现
大型语言模型虽具备强大语言能力,但在知识密集型推理任务中表现不佳。知识库问答(KBQA)因需多跳推理而尤为挑战,现有方法依赖预设流水线逐步推理,缺乏灵活性且易产生错误累积。为此,我们提出KG-Hopper,一种基于强化学习的新框架,使紧凑型开放语言模型能在单次推理中完成整合式多跳知识图谱推理。与逐步推理不同,我们训练一个推理模型,将整个知识图谱遍历与决策过程嵌入统一的“思考”阶段,实现跨步骤依赖的全局推理及支持回溯的动态路径探索。在八个知识图谱推理基准上的实验表明,基于70亿参数模型的KG-Hopper,持续优于最大达700亿参数的多步系统,并达到GPT-3.5-Turbo与GPT-4o-mini等专有模型的竞争力,同时保持轻量、开源与数据高效。代码已公开于:https://github.com/Wangshuaiia/KG-Hopper。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate impressive natural language capabilities but often struggle with knowledge-intensive reasoning tasks. Knowledge Base Question Answering (KBQA), which leverages structured Knowledge Graphs (KGs) exemplifies this challenge due to the need for accurate multi-hop reasoning. Existing approaches typically perform sequential reasoning steps guided by predefined pipelines, restricting flexibility and causing error cascades due to isolated reasoning at each step. To address these limitations, we propose KG-Hopper, a novel Reinforcement Learning (RL) framework that empowers compact open LLMs with the ability to perform integrated multi-hop KG reasoning within a single inference round. Rather than reasoning step-by-step, we train a Reasoning LLM that embeds the entire KG traversal and decision process into a unified ``thinking'' stage, enabling global reasoning over cross-step dependencies and dynamic path exploration with backtracking. Experimental results on eight KG reasoning benchmarks show that KG-Hopper, based on a 7B-parameter LLM, consistently outperforms larger multi-step systems (up to 70B) and achieves competitive performance with proprietary models such as GPT-3.5-Turbo and GPT-4o-mini, while remaining compact, open, and data-efficient. The code is publicly available at: https://github.com/Wangshuaiia/KG-Hopper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。