用可微图学习层替代分类头,让模型利用样本间关系提升性能。
GLL: A Differentiable Graph Learning Layer for Neural Networks
- 提出可微图学习层,端到端融合相似性图构建与标签传播。
- 在多个数据集上实现更平滑的标签过渡和更强的抗攻击能力。
- 适合需要关系建模的分类任务,尤其对对抗样本敏感场景。
标准深度学习分类架构通过投影头和Softmax函数生成标签预测,但忽略了样本间的关联信息。近期研究尝试将基于图的学习方法(如拉普拉斯学习)与神经网络结合,用于监督和半监督学习,但以往工作要么近似梯度,要么分离处理过程,未能实现端到端集成。本文通过伴随法推导出通用图学习层的反向传播公式,提出一种新型可微图学习层(GLL),可精确整合相似性图构建与图拉普拉斯标签传播,作为通用分类任务中投影头和Softmax函数的替代。实验表明,该方法在数据间实现平滑标签过渡,提升泛化能力与对抗攻击鲁棒性,并改善训练动态,优于传统Softmax方法。
原文摘要 · Abstract (English)
Standard deep learning architectures used for classification generate label predictions with a projection head and softmax activation function. Although successful, these methods fail to leverage the relational information between samples for generating label predictions. In recent works, graph-based learning techniques, namely Laplace learning, have been heuristically combined with neural networks for both supervised and semi-supervised learning (SSL) tasks. However, prior works approximate the gradient of the loss function with respect to the graph learning algorithm or decouple the processes; end-to-end integration with neural networks is not achieved. In this work, we derive backpropagation equations, via the adjoint method, for inclusion of a general family of graph learning layers into a neural network. The resulting method, distinct from graph neural networks, allows us to precisely integrate similarity graph construction and graph Laplacian-based label propagation into a neural network layer, replacing a projection head and softmax activation function for general classification task. Our experimental results demonstrate smooth label transitions across data, improved generalization and robustness to adversarial attacks, and improved training dynamics compared to a standard softmax-based approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。