让大模型像查资料一样推理,用知识图谱提升准确性。
GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation
- 结合参数记忆与外部知识图谱,分观察、反思、输出三步推理。
- 小模型用GIVE后表现超越大模型,科学任务准确率提升至88.2%。
- 无需训练即可应对新问题,适合科研及冷门领域应用。
现有基于上下文提示或强化学习的方法依赖大模型内部知识生成可靠思维链,但单次前向传播仍无法解决某些问题。而基于代理的推理系统需依赖全面的非参数知识库,这在科学和专业领域往往成本过高或不可行。本文提出一种新型推理方法GIVE(Graph Inspired Veracity Extrapolation),将参数化记忆与非参数记忆融合,以最小外部输入提升推理准确性。GIVE引导大模型代理执行三步:选择最相关专家数据(observe)、进行特定查询的发散思考(reflect)、整合信息生成最终输出(speak)。大量实验表明:(1) GIVE可提升各尺寸大模型性能;(2) 在某些场景下,GPT3.5T+GIVE的表现优于GPT4;(3) 在科学与开放域评估中均有效;(4) 该方法无需训练,使大模型可处理超出其训练数据的新问题,准确率从43.5%提升至88.2%;(5) 可适配135至840k以上节点的知识图谱,支持小规模与噪声大的知识源;(6) 推理过程完全可解释。
原文摘要 · Abstract (English)
Existing approaches based on context prompting or reinforcement learning (RL) to improve the reasoning capacities of large language models (LLMs) depend on the LLMs' internal knowledge to produce reliable Chain-Of-Thought (CoT). However, no matter the size of LLMs, certain problems cannot be resolved in a single forward pass. Meanwhile, agent-based reasoning systems require access to a comprehensive nonparametric knowledge base, which is often costly or not feasible for use in scientific and niche domains. We present Graph Inspired Veracity Extrapolation (GIVE), a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input. GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific divergent thinking (reflect), and then synthesize this information to produce the final output (speak). Extensive experiments demonstrated the following benefits of our framework: (1) GIVE boosts the performance of LLMs across various sizes. (2) In some scenarios, GIVE allows smaller LLMs to surpass larger, more sophisticated ones in scientific tasks (GPT3.5T + GIVE > GPT4). (3) GIVE is effective on scientific and open-domain assessments. (4) GIVE is a training-free method that enables LLMs to tackle new problems that extend beyond their training data (up to 43.5% -> 88.2%} accuracy improvement). (5) GIVE allows LLM agents to reason using both restricted (very small) and noisy (very large) knowledge sources, accommodating knowledge graphs (KG) ranging from 135 to more than 840k nodes. (6) The reasoning process involved in GIVE is fully interpretable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。