用大模型提升动态文本图的表示能力,让图神经网络更好捕捉时空文本信息。
LLM-driven Knowledge Distillation for Dynamic Text-Attributed Graphs
- 通过轻量GNN从大模型提取的文本表示中蒸馏知识,融合时空结构信息。
- 在6个真实数据集上,链接预测和边分类任务性能显著优于基线模型。
- 适合处理社交、引用等含时间与文本的动态图场景,如未来关系预测。
动态文本属性图(DyTAGs)在社交、合作、引用、通信和评论网络等场景中有广泛应用。这些网络中节点和边常带有文本描述,且图结构随时间演化。未来链接预测、边分类、关系生成等下游任务需要能编码结构、时间与文本信息的强表示。尽管图神经网络(GNN)擅长处理结构化数据,但动态图中的时序信息建模仍具挑战。本文提出面向动态文本属性图的LLM驱动知识蒸馏方法(LKD4DyTAG),引入时序编码以实现图卷积同时捕获时序与结构信息。通过简单而有效的方式对边进行时序编码,使隐藏表示同时包含时间与结构特征。为利用大模型(LLM)的文本处理能力学习更丰富的表示,我们基于邻域文本属性生成的LLM边表示,通过轻量级GNN将知识蒸馏至时空表示中。知识蒸馏目标促使GNN学习更有效地整合结构、时序与文本信息。我们在六个真实世界DyTAG数据集上进行了大量实验,验证了该方法在未来链接预测与边分类任务上的有效性。结果表明,相比基线模型,本方法显著提升了下游任务性能。
原文摘要 · Abstract (English)
Dynamic Text-Attributed Graphs (DyTAGs) have numerous real-world applications, e.g. social, collaboration, citation, communication, and review networks. In these networks, nodes and edges often contain text descriptions, and the graph structure can evolve over time. Future link prediction, edge classification, relation generation, and other downstream tasks on DyTAGs require powerful representations that encode structural, temporal, and textual information. Although graph neural networks (GNNs) excel at handling structured data, encoding temporal information within dynamic graphs remains a significant challenge. In this work, we propose LLM-driven Knowledge Distillation for Dynamic Text Attributed Graph (LKD4DyTAG) with temporal encoding to address these challenges. We use a simple, yet effective approach to encode temporal information in edges so that graph convolution can simultaneously capture both temporal and structural information in the hidden representations. To leverage LLM's text processing capabilities for learning richer representations on DyTAGs, we distill knowledge from LLM-driven edge representations (based on a neighborhood's text attributes) into saptio-temporal representations using a lightweight GNN model that encodes temporal and structural information. The objective of knowledge distillation enables the GNN to learn representations that more effectively encode the available structural, temporal, and textual information in DyTAG. We conducted extensive experimentation on six real-world DyTAG datasets to verify the effectiveness of our approach LKD4DyTAG for future link prediction and edge classification task. The results show that our approach significantly improves the performance of downstream tasks compared to the baseline models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。