用自注意力动态采样,让轻量KAN模型超越GNN的图数据表现。
Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Sampling
- 用自注意力机制动态选关键节点,提升学生模型对图结构的感知能力。
- 在6个真实数据集上,性能超3个GNN教师模型3.05%-3.62%。
- 参数减少16.69倍,每轮训练快55.75%,适合边缘部署。
图神经网络(GNN)在建模复杂图结构数据方面取得成功,但其高计算与内存开销限制了在资源受限的边缘设备上的部署。将GNN知识蒸馏到MLP虽能减轻负担,但MLP因固定激活函数和缺乏邻域聚合,性能受限。为此,我们提出SA-DSD框架,首次将增强型科尔莫戈罗夫-阿诺德网络(KAN)作为学生模型。通过引入可学习频率基、相位偏移及优化算法,改进傅里叶型KAN(FR-KAN+),显著提升非线性拟合能力且保持低计算复杂度。为弥补学生模型缺少邻域聚合的缺陷,SA-DSD利用自注意力机制动态识别关键节点,构建自适应采样概率矩阵,并强制师生预测一致性。在6个真实世界数据集上的大量实验表明,在归纳与多数归纳设置下,SA-DSD比3个GNN教师模型高出3.05%-3.62%,较FR-KAN+提升15.61%。同时,相比基准方法,参数量减少16.69倍,每轮平均运行时间降低55.75%。
原文摘要 · Abstract (English)
Recent success of graph neural networks (GNNs) in modeling complex graph-structured data has fueled interest in deploying them on resource-constrained edge devices. However, their substantial computational and memory demands present ongoing challenges. Knowledge distillation (KD) from GNNs to MLPs offers a lightweight alternative, but MLPs remain limited by fixed activations and the absence of neighborhood aggregation, constraining distilled performance. To tackle these intertwined limitations, we propose SA-DSD, a novel self-attention-guided dynamic sampling distillation framework. To the best of our knowledge, this is the first work to employ an enhanced Kolmogorov-Arnold Network (KAN) as the student model. We improve Fourier KAN (FR-KAN+) with learnable frequency bases, phase shifts, and optimized algorithms, substantially improving nonlinear fitting capability over MLPs while preserving low computational complexity. To explicitly compensate for the absence of neighborhood aggregation that is inherent to both MLPs and KAN-based students, SA-DSD leverages a self-attention mechanism to dynamically identify influential nodes, construct adaptive sampling probability matrices, and enforce teacher-student prediction consistency. Extensive experiments on six real world datasets demonstrate that, under inductive and most of transductive settings, SA-DSD surpasses three GNN teachers by 3.05%-3.62% and improves FR-KAN+ by 15.61%. Moreover, it achieves a 16.69x parameter reduction and a 55.75% decrease in average runtime per epoch compared to key benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。