用结果监督训练大模型,让其自主推理问答知识库问题。
KnowCoder-A1: Incentivizing Agentic Reasoning Capability with Outcome Supervision for KBQA
- 通过结果导向的强化学习,激励模型自主探索推理路径。
- 在三个主流数据集上表现优于现有方法,零样本下提升11.1%。
- 仅用1/12数据量就达到顶尖性能,适合资源受限场景。
知识库问答(KBQA)旨在基于结构化知识库回答自然语言问题。近期工作采用代理式推理范式,让大语言模型(LLM)迭代分解问题、生成逻辑查询并交互知识库获取答案。然而,这些方法通常在过程监督下微调模型,激励不足,难以增强代理推理能力。本文提出KnowCoder-A1,一种能自主执行知识库代理推理的大模型。为激励自主探索,它采用多阶段课程强化学习,在仅结果监督下训练。首先,利用基于结果的拒绝采样获得少量高质量推理轨迹进行微调;随后,通过从易到难的课程设计缓解结果监督下的奖励稀疏性。在仅结果监督下训练的KnowCoder-A1展现出强大推理行为,在三个主流数据集上持续领先。尤其在GrailQA的零样本子集上,相对提升达11.1%,且仅使用了1/12的训练数据,充分体现了其强大的代理推理能力。
原文摘要 · Abstract (English)
Knowledge Base Question Answering (KBQA) aims to answer natural-language questions over a structured Knowledge Base (KB). Recent work improves KBQA by adopting an agentic reasoning paradigm, in which Large Language Models (LLMs) iteratively decompose a question, generate its corresponding logical queries, and interact with the KB to derive the answer. However, these methods typically fine-tune LLMs on reasoning trajectories synthesized via process supervision, which offers weak incentives for exploration and thus fails to strengthen the agentic reasoning ability. In this paper, we propose KnowCoder-A1, an LLM that can autonomously perform agentic reasoning on KBs to obtain answers. To incentivize autonomous exploration, KnowCoder-A1 trains the LLM under outcome-only supervision via a multi-stage curriculum reinforcement learning with an easy-to-hard curriculum. To establish foundational agentic capabilities, KnowCoder-A1 first fine-tunes the LLM on a small set of high-quality trajectories obtained through outcome-based rejection sampling. Then, to alleviate the reward sparsity inherent in outcome-only supervision, it applies multi-stage curriculum RL with reward schedules that progress from easy to hard. Trained with outcome-only supervision, KnowCoder-A1 exhibits powerful reasoning behaviors and consistently outperforms prior approaches across three mainstream datasets. Notably, on the zero-shot subset of GrailQA, KnowCoder-A1 achieves up to an 11.1% relative improvement while using only one-twelfth of the training data, demonstrating strong agentic reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。