用掩码语言模型统一图任务,让模型跨域零样本泛化。
Masked Language Models are Good Heterogeneous Graph Generalizers
- 用元路径生成文本序列替代节点嵌入,统一图任务为填空预测
- 在四个数据集上实现少样本与零样本下优于现有方法的泛化性能
- 适合需要跨域通用图学习的研究者或工业应用
异质图神经网络(HGNNs)擅长捕捉异质图(HGs)中的结构与语义信息,但在跨领域和跨任务上的泛化能力不足。随着大语言模型(LLMs)的发展,有研究尝试将HGNNs与LLMs结合以提升泛化性,但通常通过HGNN编码结构信息为图标记(HG tokens),导致与LLM嵌入空间不一致,影响理解;且这些标记多源于节点级任务,限制了任务泛化能力。为此,我们提出一种基于掩码语言模型的方法MLM4HG:用元路径构建文本序列替代传统图标记,设计定制化文本模板,将不同图任务统一为连贯的填空式掩码预测范式。具体地,首先将多源异质图转换为基于元路径的文本,再与统一的任务文本合并形成图基语料库,输入预训练语言模型进行微调,并限定目标词汇表。大量跨域与多任务实验表明,在四种真实世界数据集上,MLM4HG在少样本与零样本场景下均显著优于现有方法。代码已开源。
原文摘要 · Abstract (English)
Heterogeneous graph neural networks (HGNNs) excel at capturing structural and semantic information in heterogeneous graphs (HGs), while struggling to generalize across domains and tasks. With the rapid advancement of large language models (LLMs), a recent study explored the integration of HGNNs with LLMs for generalizable heterogeneous graph learning. However, this approach typically encodes structural information as HG tokens using HGNNs, and disparities in embedding spaces between HGNNs and LLMs have been shown to bias the LLM's comprehension of HGs. Moreover, since these HG tokens are often derived from node-level tasks, the model's ability to generalize across tasks remains limited. To this end, we propose a simple yet effective Masked Language Modeling-based method, called MLM4HG. MLM4HG introduces metapath-based textual sequences instead of HG tokens to extract structural and semantic information inherent in HGs, and designs customized textual templates to unify different graph tasks into a coherent cloze-style 'mask' token prediction paradigm. Specifically,MLM4HG first converts HGs from various domains to texts based on metapaths, and subsequently combines them with the unified task texts to form a HG-based corpus. Moreover, the corpus is fed into a pretrained LM for fine-tuning with a constrained target vocabulary, enabling the fine-tuned LM to generalize to unseen target HGs. Extensive cross-domain and multi-task experiments on four real-world datasets demonstrate the superior generalization performance of MLM4HG over state-of-the-art methods in both few-shot and zero-shot scenarios. Our code is available at https://github.com/BUPT-GAMMA/MLM4HG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。