arXiv:2506.07408cs.LGcs.AI2025-06被引 1

提出分数阶雅可比矩阵微分法,让神经网络训练更高效。

Fractional-order Jacobian Matrix Differentiation and Its Application in Artificial Neural Networks

  • 基于整数阶雅可比矩阵定义分数阶微分,支持自动微分。
  • 在多层感知机中替换线性层,测试集指标显著提升。
  • 适合研究深度学习优化与分数阶微积分的学者使用。

分数阶微分与整数阶微分具有不同特性,可提升神经网络优化效果。然而,由于理论研究不足,目前尚无与自动微分(Autograd)完全兼容的分数阶矩阵微分方法。为此,本文提出一种分数阶雅可比矩阵微分方法(${\bf{J}^α}$),基于整数阶雅可比矩阵定义,实现基于矩阵的分数阶链式法则。结合线性模块与分数阶微分,设计了分数阶Autograd技术,使分数阶微分可在隐藏层中应用,显著增强其在深度学习中的实用性。实验基于PyTorch框架,构建分数阶线性层(FLinear)并替换MLP中的nn.Linear。通过训练集与验证集损失的定性分析、测试集指标的定量评估,以及训练时的时间消耗和GPU内存使用分析,验证了${\bf{J}^α}$的优越性能,证明其是深度学习领域优秀的分数阶梯度下降方法。

原文摘要 · Abstract (English)

Fractional-order differentiation has many characteristics different from integer-order differentiation. These characteristics can be applied to the optimization algorithms of artificial neural networks to obtain better results. However, due to insufficient theoretical research, at present, there is no fractional-order matrix differentiation method that is perfectly compatible with automatic differentiation (Autograd) technology. Therefore, we propose a fractional-order matrix differentiation calculation method. This method is introduced by the definition of the integer-order Jacobian matrix. We denote it as fractional-order Jacobian matrix differentiation (${\bf{J}^α}$). Through ${\bf{J}^α}$, we can carry out the matrix-based fractional-order chain rule. Based on the Linear module and the fractional-order differentiation, we design the fractional-order Autograd technology to enable the use of fractional-order differentiation in hidden layers, thereby enhancing the practicality of fractional-order differentiation in deep learning. In the experiment, according to the PyTorch framework, we design fractional-order Linear (FLinear) and replace nn.Linear in the multilayer perceptron with FLinear. Through the qualitative analysis of the training set and validation set $Loss$, the quantitative analysis of the test set indicators, and the analysis of time consumption and GPU memory usage during model training, we verify the superior performance of ${\bf{J}^α}$ and prove that it is an excellent fractional-order gradient descent method in the field of deep learning.

神经网络分数阶微分自动微分深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。