用知识蒸馏让Transformer学会图结构先验，提升建模能力
Enhancing Transformer with GNN Structural Knowledge via Distillation: A Novel Approach
- 通过多尺度结构知识蒸馏，将GNN的图结构先验注入Transformer
- 提出微观-宏观联合损失，有效对齐GNN与Transformer的特征表示
- 适合研究图神经网络与Transformer融合的学者，尤其关注结构先验迁移
将图神经网络（GNN）的结构归纳偏置与Transformer的全局上下文建模能力相结合，是图表示学习中的关键挑战。尽管GNN通过消息传递机制擅长捕捉局部拓扑模式，但其在建模长程依赖和可并行性方面的局限性限制了其在大规模场景中的应用。相反，Transformer利用自注意力机制实现全局感受野，却难以继承GNN的内在图结构先验。本文提出一种新颖的知识蒸馏框架，系统性地将多尺度结构知识从GNN教师模型转移到Transformer学生模型，为跨架构蒸馏提供了新思路。该框架通过微-宏观蒸馏损失和多尺度特征对齐，有效弥合了GNN与Transformer之间的架构鸿沟。本工作确立了一种在Transformer架构中继承图结构偏置的新范式，具有广泛的应用前景。
原文摘要 · Abstract (English)
Integrating the structural inductive biases of Graph Neural Networks (GNNs) with the global contextual modeling capabilities of Transformers represents a pivotal challenge in graph representation learning. While GNNs excel at capturing localized topological patterns through message-passing mechanisms, their inherent limitations in modeling long-range dependencies and parallelizability hinder their deployment in large-scale scenarios. Conversely, Transformers leverage self-attention mechanisms to achieve global receptive fields but struggle to inherit the intrinsic graph structural priors of GNNs. This paper proposes a novel knowledge distillation framework that systematically transfers multiscale structural knowledge from GNN teacher models to Transformer student models, offering a new perspective on addressing the critical challenges in cross-architectural distillation. The framework effectively bridges the architectural gap between GNNs and Transformers through micro-macro distillation losses and multiscale feature alignment. This work establishes a new paradigm for inheriting graph structural biases in Transformer architectures, with broad application prospects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。