用扩散模型辅助蒸馏，让MLP更好学图自监督表示
Diffusion-Assisted Distillation for Self-Supervised Graph Representation Learning with MLPs
- 用去噪扩散模型做教师助手，帮MLP从GNN学习
- 在多个数据集上超越现有蒸馏方法，性能提升显著
- 适合追求轻量高效图表示的工程应用
大规模应用中，人们越来越倾向于用轻量级多层感知机（MLPs）替代图神经网络（GNNs），通过知识蒸馏实现。然而，在自监督图表示学习中将GNN知识蒸馏到MLP更具挑战性，因为自监督性能更依赖于模型的归纳偏置。为此，本文提出扩散辅助自监督图表示学习的MLP蒸馏方法（DAD-SGM）。该方法引入去噪扩散模型作为教师助手，更有效地将教师GNN的知识传递给学生MLP，提升了MLP在自监督学习中的泛化性和鲁棒性。大量实验表明，相比现有最优蒸馏方法，DAD-SGM能更高效地蒸馏自监督GNN的知识。代码已公开于https://github.com/SeongJinAhn/DAD-SGM。
原文摘要 · Abstract (English)
For large-scale applications, there is growing interest in replacing Graph Neural Networks (GNNs) with lightweight Multi-Layer Perceptrons (MLPs) via knowledge distillation. However, distilling GNNs for self-supervised graph representation learning into MLPs is more challenging. This is because the performance of self-supervised learning is more related to the model's inductive bias than supervised learning. This motivates us to design a new distillation method to bridge a huge capacity gap between GNNs and MLPs in self-supervised graph representation learning. In this paper, we propose \textbf{D}iffusion-\textbf{A}ssisted \textbf{D}istillation for \textbf{S}elf-supervised \textbf{G}raph representation learning with \textbf{M}LPs (DAD-SGM). The proposed method employs a denoising diffusion model as a teacher assistant to better distill the knowledge from the teacher GNN into the student MLP. This approach enhances the generalizability and robustness of MLPs in self-supervised graph representation learning. Extensive experiments demonstrate that DAD-SGM effectively distills the knowledge of self-supervised GNNs compared to state-of-the-art GNN-to-MLP distillation methods. Our implementation is available at https://github.com/SeongJinAhn/DAD-SGM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。