用李雅普诺夫指数选超参，让深度网络训练更快更稳。
Utilizing Lyapunov Exponents in designing deep neural networks
- 通过计算激活函数的李雅普诺夫指数，指导超参数选择。
- 负指数越大，模型收敛越快，训练更稳定。
- 适合关注训练效率与优化过程的深度学习研究者。
训练大型深度神经网络资源消耗大。本研究探讨李雅普诺夫指数是否能加速该过程，通过在不同激活函数的隐藏层中构建神经网络，并使用不同随机种子初始化模型权重，结合传统梯度下降法计算李雅普诺夫指数。结果表明，学习率的变化可引发模型权重的混沌行为；具有更负李雅普诺夫指数的激活函数展现出更好的收敛性。此外，研究还证明李雅普诺夫指数可用于选择有效的初始权重，从而提升优化效率。
原文摘要 · Abstract (English)
Training large deep neural networks is resource intensive. This study investigates whether Lyapunov exponents can accelerate this process by aiding in the selection of hyperparameters. To study this I formulate an optimization problem using neural networks with different activation functions in the hidden layers. By initializing model weights with different random seeds, I calculate the Lyapunov exponent while performing traditional gradient descent on these model weights. The findings demonstrate that variations in the learning rate can induce chaotic changes in model weights. I also show that activation functions with more negative Lyapunov exponents exhibit better convergence properties. Additionally, the study also demonstrates that Lyapunov exponents can be utilized to select effective initial model weights for deep neural networks, potentially enhancing the optimization process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。