让大模型直接学会图结构知识,推理时无需再看原图。
GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Models
- 通过精细微调将图结构知识内化到模型参数中
- 在超大规模图上表现优于基线,小图推理成本更低
- 仅用轻量LoRA模块存储知识,适合部署于资源受限场景
大语言模型在序列文本建模和跨任务泛化方面表现出色,但有效适配结构化数据(如知识图谱或网络图)仍是根本挑战。现有方法多将图转为文本序列,导致令牌开销过大;或引入额外模块将图编码为固定长度的表示,需大规模图-文语料后训练和复杂对齐流程,且常因模态对齐不佳导致性能欠佳。本文提出GRIP,不依赖复杂的图序列化或专用编码模块,而是通过精心设计的微调任务,将复杂关系知识直接内化至大模型参数中。所获结构知识以轻量级LoRA模块形式存储,使微调后的模型可在无需原始图作为上下文的情况下执行多种图相关任务。大量实验验证:对于超出模型上下文窗口的大型图,GRIP持续优于基线;对小规模图,性能相当但推理成本显著降低。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities in modeling sequential textual data and generalizing across diverse tasks. However, effectively adapting LLMs to structural data, such as knowledge graphs or web graphs, remains a fundamental challenge. Some approaches adopt complex strategies to convert graphs into text sequences, resulting in significant token overhead and rendering them impractical for large-scale graphs. Others introduce additional modules to encode graphs into fixed-size token representations for LLMs. However, these methods typically require large-scale post-training on graph-text corpus and complex alignment procedures, yet often yield sub-optimal results due to poor modality alignment. In this work, we propose GRIP. Instead of relying on heavy graph serialization or specialized graph encoding modules, GRIP directly internalizes complex relational knowledge from graphs into the parameters of LLM through carefully designed fine-tuning tasks. The acquired structural knowledge is compactly stored in lightweight LoRA modules, enabling the fine-tuned LLM to perform a wide range of tasks over the internalized graph without requiring access to the original graph as context at inference time. Extensive experiments validate our approach. For graphs that cannot fit within the LLMs context window, GRIP consistently outperforms LLM baselines by leveraging internalized graph knowledge, while for small-scale graphs, it achieves comparable performance with substantially lower inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。