通过优化负曲率提升双曲神经网络泛化能力,解决传统方法因曲率不当导致性能下降的问题。
Curvature Learning for Generalization of Hyperbolic Neural Networks
- 提出基于尖锐度的曲率学习方法,通过双层优化平滑损失曲面
- 在四类任务中验证,显著提升双曲神经网络的分类与泛化性能
- 适合处理层次结构数据的模型优化,尤其适用于小样本与噪声数据场景
双曲神经网络(HNNs)利用负曲率空间的几何特性,在表示具有层次结构的真实世界数据方面表现出色。曲率在优化HNNs中起关键作用,不合适的曲率可能导致收敛至次优参数,降低整体性能。目前尚缺乏曲率对HNNs影响的理论基础。本文推导了HNNs的PAC-Bayesian泛化界,揭示了曲率通过影响损失曲面平滑性来决定泛化能力的作用机制。基于该理论,我们提出一种尖锐度感知的曲率学习方法,通过双层优化最小化曲率的尖锐度度量,并设计隐式微分算法高效求解梯度。我们给出了近似误差和收敛性分析,证明近似误差有上界,且方法能通过约束梯度实现收敛。在四类设置——分类、长尾数据学习、噪声数据学习和少样本学习——上的实验表明,该方法可有效提升HNNs性能。
原文摘要 · Abstract (English)
Hyperbolic neural networks (HNNs) have demonstrated notable efficacy in representing real-world data with hierarchical structures via exploiting the geometric properties of hyperbolic spaces characterized by negative curvatures. Curvature plays a crucial role in optimizing HNNs. Inappropriate curvatures may cause HNNs to converge to suboptimal parameters, degrading overall performance. So far, the theoretical foundation of the effect of curvatures on HNNs has not been developed. In this paper, we derive a PAC-Bayesian generalization bound of HNNs, highlighting the role of curvatures in the generalization of HNNs via their effect on the smoothness of the loss landscape. Driven by the derived bound, we propose a sharpness-aware curvature learning method to smooth the loss landscape, thereby improving the generalization of HNNs. In our method, we design a scope sharpness measure for curvatures, which is minimized through a bi-level optimization process. Then, we introduce an implicit differentiation algorithm that efficiently solves the bi-level optimization by approximating gradients of curvatures. We present the approximation error and convergence analyses of the proposed method, showing that the approximation error is upper-bounded, and the proposed method can converge by bounding gradients of HNNs. Experiments on four settings: classification, learning from long-tailed data, learning from noisy data, and few-shot learning show that our method can improve the performance of HNNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。