让图语言模型用少量标注数据快速适配新任务,准确率更高、推理更快。
GraphLAMA: Enabling Efficient Adaptation of Graph Language Models with Limited Annotations
- 用图神经网络将节点映射到语言模型的词向量空间,实现图文混合表示。
- 仅需少量标注样本微调,准确率比传统方法提升4.91%。
- 适合标注稀缺但需快速部署的图分析场景,如医疗或金融风控。
大型语言模型(LLMs)在多个领域展现出强大能力,并被引入用于图分析,形成图语言模型(GLMs)。部分GLMs通过自然语言描述任务并利用提示中的少量示例进行零参数调整,称为上下文学习(ICL)。另一些则依赖大量标注数据进行指令微调以提升性能。然而,我们指出图上的ICL存在固定参数导致效果不佳、长上下文影响效率的问题;而指令微调又需要大量标签,在真实场景中难以获取。为此,本文提出GraphLAMA方法,增加一个高效参数适配阶段,仅用少量标注即可快速适配未见图与任务,在提升预测精度的同时加快推理速度。具体地,模型骨干采用设计精良的图神经网络(GNN),将节点映射至LLM的词向量空间,任务指令以节点与语言令牌的混合形式表示。预训练阶段仅训练除LLM外的参数,捕捉通用知识;适配阶段仅基于少量样本更新部分预训练参数。在少样本/零样本节点分类与摘要生成任务上,GraphLAMA达到当前最优性能,准确率绝对提升4.91%。相较于ICL,5-shot设置下推理速度提升10倍。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated their strong capabilities in various domains, and have been recently integrated for graph analysis as graph language models (GLMs). With LLMs as the predictor, some GLMs can interpret unseen tasks described by natural language, and learn from a few examples in the prompts without parameter tuning, known as in-context learning (ICL). Another subset of GLMs utilizes abundant training labels to enhance model performance, known as instruction tuning. However, we argue that ICL on graphs has effectiveness issues due to fixed parameters and efficiency issues due to long context. Meanwhile, the large amount of labeled data required for instruction tuning can be difficult to obtain in real-world scenarios. To this end, we aim to introduce an extra parameter adaptation stage that can efficiently tailor GLMs to an unseen graph and task with only a few labeled examples, in exchange for better prediction accuracy and faster inference speed. For implementation, in this paper we propose GraphLAMA method, with its model backbone and learning schemes specialized for efficient tuning and inference. Specifically, for model backbone, we use a graph neural network (GNN) with several well-designed components to transform nodes into the representation space of LLM tokens. Task instructions can then be represented as a mixture of node and language tokens. In the pre-training stage, model parameters except the LLM will be trained with different tasks to capture general knowledge. In the adaptation stage, only a few pre-trained parameters will be updated based on few-shot examples. Extensive experiments on few/zero-shot node classification and summary generation show that our proposed GraphLAMA achieves state-of-the-art performance with 4.91% absolution improvement in accuracy. Compared with ICL, our inference speed can be 10 times faster under 5-shot setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。