arXiv:2507.17168cs.AI2025-07

用图推理任务训练大模型,提升其解决新问题的能力。

Improving LLMs' Generalized Reasoning Abilities by Graph Problems

  • 用图问题数据进行持续预训练,教会模型多样推理模式。
  • 在数学推理上最高提升4.9%,非数学推理最高提升21.2%。
  • 适合想提升模型泛化推理能力的研究者与工程师。

大语言模型在推理任务上取得显著进展,但在新颖复杂问题上表现仍不足。针对数学推理的领域特定持续预训练方法虽有效,但难以迁移至更广泛的推理任务。本文首次提出利用图问题推理(GPR)来增强大模型的通用推理能力。GPR涵盖路径查找、网络分析、数值计算和拓扑推理,需复杂逻辑与关系推理,适合作为训练多样推理模式的材料。为此,我们构建了首个专用于GPR的持续预训练语料库GraphPile,包含109亿词元,覆盖23种图任务,涵盖思维链、程序链、执行轨迹及真实世界图数据。基于GraphPile,我们在Llama 3、Llama 3.1和Gemma 2等主流基础模型上训练出GraphMind,使数学推理准确率最高提升4.9%,非数学推理(如逻辑与常识推理)最高提升21.2%。本工作首次将图问题推理用于提升模型通用推理能力,并提供首个同类数据集,弥合了领域特定预训练与通用推理之间的差距,推动大模型的适应性与鲁棒性发展。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have made remarkable strides in reasoning tasks, yet their performance often falters on novel and complex problems. Domain-specific continued pretraining (CPT) methods, such as those tailored for mathematical reasoning, have shown promise but lack transferability to broader reasoning tasks. In this work, we pioneer the use of Graph Problem Reasoning (GPR) to enhance the general reasoning capabilities of LLMs. GPR tasks, spanning pathfinding, network analysis, numerical computation, and topological reasoning, require sophisticated logical and relational reasoning, making them ideal for teaching diverse reasoning patterns. To achieve this, we introduce GraphPile, the first large-scale corpus specifically designed for CPT using GPR data. Spanning 10.9 billion tokens across 23 graph tasks, the dataset includes chain-of-thought, program-of-thought, trace of execution, and real-world graph data. Using GraphPile, we train GraphMind on popular base models Llama 3 and 3.1, as well as Gemma 2, achieving up to 4.9 percent higher accuracy in mathematical reasoning and up to 21.2 percent improvement in non-mathematical reasoning tasks such as logical and commonsense reasoning. By being the first to harness GPR for enhancing reasoning patterns and introducing the first dataset of its kind, our work bridges the gap between domain-specific pretraining and universal reasoning capabilities, advancing the adaptability and robustness of LLMs.

图推理大模型持续预训练通用推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。