arXiv:2505.15062cs.CLcs.AI2025-05被引 9

小模型通过强化学习学会自主推理新知识,效率远超大模型。

SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learning

  • 用工具增强的强化学习训练小模型,分三步检索并构建新知识
  • 在生物医学和常识任务上准确率分别达75.4%和81.3%,超越GPT-3.5
  • 仅用不到10%的令牌消耗,适合资源受限场景

知识外推是通过组合和扩展已有显式知识来推断新信息的过程,对在难以获取完整外部知识的专有领域解决复杂问题至关重要。我们提出SAKE(结构化智能体知识外推),一种基于强化学习的智能体框架,使大语言模型能自主通过工具增强的强化学习检索并外推结构化知识。SAKE定义了两个外部知识图谱工具:实体组构建和跨组三元组检索。模型在三轮推理中学习交替使用这两个工具:提取关键实体、过滤相关概念组,并通过类比构建新三元组进行关联推理。整个流程通过GRPO端到端优化,奖励函数结合输出格式与答案正确性,教会模型该检索什么以及如何推理。实验表明,微调后的Qwen2.5-7B模型在生物医学(75.4% vs. 70.1%)和常识(81.3% vs. 74.7%)基准上超越GPT-3.5-Turbo,同时令牌消耗降低90%以上。结果表明,基于不完整结构化知识的关联推理无需大型模型或复杂多步提示,可通过强化学习与合适工具,由小型开源模型端到端学习实现。代码已开源。

原文摘要 · Abstract (English)

Knowledge extrapolation is the process of inferring novel information by combining and extending existing knowledge that is explicitly available. It is essential for solving complex questions in specialized domains where retrieving comprehensive external knowledge is impractical. We propose SAKE (Structured Agentic Knowledge Extrapolation), a RL powered agentic framework that trains LLMs to autonomously retrieve and extrapolate structured knowledge through tool-augmented reinforcement learning. SAKE defines two external KG tools: entity group construction and cross-group triplet retrieval. The model learns to interleave these 2 retrieval tools during a three-turn rollout: extracting key entities, filtering relevant concept groups, and associative reasoning by constructing new triplets through analogy. The entire pipeline is optimized end-to-end with GRPO using a reward that combines output format and answer correctness, teaching the model what to retrieve and how to reason over it. Our experiments proved that SAKE fine-tuned Qwen2.5-7B model surpasses GPT-3.5-Turbo with state-of-the-art agentic KG reasoning on both biomedical (75.4% vs. 70.1%) and commonsense (81.3% vs. 74.7%) benchmarks, while reducing token usage by over 90%. These results demonstrate that associative reasoning over incomplete structured knowledge does not require large models with complex, multi-step prompting, thus can be learned end-to-end by small, open-weight models through reinforcement learning with the right tools and training signal. Our code is available at https://github.com/jxfan99/SAKE.

知识推理强化学习小模型智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。