用强化学习训练大模型理解图结构，效果超越更大模型。
G1: Teaching LLMs to Reason on Graphs with Reinforcement Learning
- 用强化学习在合成图任务上微调大模型，提升图推理能力。
- 30亿参数模型在50个图任务上表现超720亿参数模型。
- 零样本泛化能力强，适合想提升模型图推理能力的研究者。
尽管大语言模型（LLMs）进展显著，其在图相关任务上的表现仍有限，制约了通用模型的发展。现有方法如预训练图基础模型或监督微调常受限于大规模、通用图数据稀缺。本文提出G1，通过在合成图理论任务上使用强化学习（RL），显著提升LLMs的图推理能力。为支持RL训练，我们构建了迄今最大的图推理数据集Erdõs，包含50种不同难度的图理论任务，共10万条训练数据和5千条测试数据，均源自真实图。在Erdõs上进行RL训练后，G1在图推理上取得显著提升，其微调后的30亿参数模型性能甚至超过24倍更大的Qwen2.5-72B-Instruct。RL训练模型在未见任务、领域及图编码方式下也表现出强零样本泛化能力，包括其他图理论基准以及真实世界的节点分类和链接预测任务，且不损害一般推理能力。研究结果表明，通过在图理论任务上用强化学习微调，可高效、可扩展地构建强大图推理模型，结合预训练大模型能力与自动生成的丰富合成数据，证明大模型具备潜在图理解能力，而强化学习可有效激发之。代码与数据已开源，项目地址：https://github.com/PKU-ML/G1，模型与数据集托管于Hugging Face集合：https://huggingface.co/collections/PKU-ML/g1-683d659e992794fc99618cf2。
原文摘要 · Abstract (English)
Although Large Language Models (LLMs) have demonstrated remarkable progress, their proficiency in graph-related tasks remains notably limited, hindering the development of truly general-purpose models. Previous attempts, including pretraining graph foundation models or employing supervised fine-tuning, often face challenges such as the scarcity of large-scale, universally represented graph data. We introduce G1, a simple yet effective approach demonstrating that Reinforcement Learning (RL) on synthetic graph-theoretic tasks can significantly scale LLMs' graph reasoning abilities. To enable RL training, we curate Erdõs, the largest graph reasoning dataset to date comprising 50 diverse graph-theoretic tasks of varying difficulty levels, 100k training data and 5k test data, all drived from real-world graphs. With RL on Erdõs, G1 obtains substantial improvements in graph reasoning, where our finetuned 3B model even outperforms Qwen2.5-72B-Instruct (24x size). RL-trained models also show strong zero-shot generalization to unseen tasks, domains, and graph encoding schemes, including other graph-theoretic benchmarks as well as real-world node classification and link prediction tasks, without compromising general reasoning abilities. Our findings offer an efficient, scalable path for building strong graph reasoners by finetuning LLMs with RL on graph-theoretic tasks, which combines the strengths of pretrained LLM capabilities with abundant, automatically generated synthetic data, suggesting that LLMs possess graph understanding abilities that RL can elicit successfully. Our implementation is open-sourced at https://github.com/PKU-ML/G1, with models and datasets hosted on Hugging Face collections https://huggingface.co/collections/PKU-ML/g1-683d659e992794fc99618cf2 for broader accessibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。