arXiv:2410.20356cs.LGcs.AI2024-10被引 4

用模型剪枝替代数据增强,提升图对比学习效果

Uncovering Capabilities of Model Pruning in Graph Contrastive Learning

  • 通过剪枝生成不同版本模型进行对比,而非依赖随机扰动数据
  • 在多个图分类任务上优于当前最优方法,提升显著
  • 适合追求高效且通用图表示学习的研究者

图对比学习在无标签情况下预训练图神经网络方面取得显著进展。现有主流方法依赖随机或学习生成的增强视图,导致语义失真;虽有领域知识引导的增强可缓解,但泛化能力受限。本文受剪枝后模型稀疏性表征能力强的启发,提出以不同剪枝版本的模型作为对比目标,而非依赖数据增强。理论上证明了模型剪枝相比数据增强的优势。实践中,以原始图输入,动态剪枝编码器的变换权重生成扰动模型,与原模型对比。为保持节点嵌入完整性,引入局部对比损失,有效缓解难负样本对训练的干扰。在多种图分类基准上,通过无监督和迁移学习验证,性能始终优于现有最先进方法。

原文摘要 · Abstract (English)

Graph contrastive learning has achieved great success in pre-training graph neural networks without ground-truth labels. Leading graph contrastive learning follows the classical scheme of contrastive learning, forcing model to identify the essential information from augmented views. However, general augmented views are produced via random corruption or learning, which inevitably leads to semantics alteration. Although domain knowledge guided augmentations alleviate this issue, the generated views are domain specific and undermine the generalization. In this work, motivated by the firm representation ability of sparse model from pruning, we reformulate the problem of graph contrastive learning via contrasting different model versions rather than augmented views. We first theoretically reveal the superiority of model pruning in contrast to data augmentations. In practice, we take original graph as input and dynamically generate a perturbed graph encoder to contrast with the original encoder by pruning its transformation weights. Furthermore, considering the integrity of node embedding in our method, we are capable of developing a local contrastive loss to tackle the hard negative samples that disturb the model training. We extensively validate our method on various benchmarks regarding graph classification via unsupervised and transfer learning. Compared to the state-of-the-art (SOTA) works, better performance can always be obtained by the proposed method.

图神经网络对比学习模型剪枝无监督学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。