8B模型通过强化学习内化知识图谱搜索能力,无需外部推理开销。
Search-on-Graph-R1: Training Large Language Models to Search Knowledge Graphs with Reinforcement Learning

- 用金标准SPARQL引导教师模型生成带路径的搜索轨迹
- 8B模型在三数据集上超越所有冻结大模型,CWQ性能最强
- 无需辅助模块或判别器,可跨模型迁移,减少搜索调用次数
知识图谱问答(KGQA)需要从主题实体出发,经过多个关系链到达答案。现有方法依赖前沿大模型通过检索工具探索图谱,但推理成本高。本文提出Search-on-Graph-R1(\sogrone{}),通过监督微调(SFT)和强化学习(RL)将导航能力内化至紧凑的8B模型中。核心思想是为每个问题构建一个由金标准SPARQL驱动的教师模型,使其借助实时的\texttt{Search}工具沿已知答案路径行进,而非自主发现路径。由于每次调用均对接真实Freebase服务器,生成的轨迹天然具备图谱依据。在WebQSP、CWQ和GrailQA上,\sogrone{}(8B)超越所有对比的冻结前沿模型,在CWQ上达到当前最优结果。推理时无需额外模块,训练中也不需大模型判别器。消融实验表明,SFT与RL贡献互补,该方法可跨模型家族迁移,且RL阶段显著减少\texttt{Search}调用次数。
原文摘要 · Abstract (English)
Knowledge graph question answering (KGQA) requires navigating from topic entities to an answer several relations away. Recent methods prompt a frontier LLM to explore the graph through a retrieval tool, but their reliance on frontier-scale inference makes them costly to deploy. We present Search-on-Graph-R1 (\sogrone{}), which internalizes this navigation into a compact 8B model through supervised fine-tuning (SFT) followed by reinforcement learning (RL). Our central idea is to scaffold a frontier teacher with each question's gold SPARQL query, so the teacher traverses a known answer-bearing path with a live \texttt{Search} tool rather than having to discover the path itself. Since every call executes against a live Freebase server, the resulting trajectories are grounded in the knowledge graph by construction. On WebQSP, CWQ, and GrailQA, \sogrone{} at 8B surpasses every frozen frontier-LLM system in our comparison and posts the strongest results on CWQ of any system we compare against. It does so using no auxiliary module at inference and no LLM judge during training. Isolating each training stage shows that SFT and RL contribute complementary gains, our approach transfers across model families, and RL learns to reach answers in fewer \texttt{Search} calls than its SFT initialization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。