深度线性网络的最优学习率需随数据调整，否则无法跨深度迁移。
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
- 基于数据设计逐层学习率，而非固定规则。
- 新方法使收敛速度在所有深度下保持恒定。
- 适用于带残差连接的深层网络，适合研究优化机制者。
本文研究深度标量线性网络 $f(x) = \prod_{l=1}^L w_l x$ 的梯度下降动态，该模型对任意整数深度均有精确的时间演化解。我们发现，即使在这一最简模型中，最优的逐层学习率缩放仍依赖于数据；而与数据无关的缩放规则无法在不同深度间迁移。采用数据依赖的最优缩放后，学习动态完全独立于数据，且仅弱依赖于深度，从而在所有深度（包括无限深度）均实现恒定的线性收敛速率。进一步分析表明，类似的数据依赖效应也存在于含残差连接的深层标量线性网络中。
原文摘要 · Abstract (English)
In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling rules fail to transfer across depths. Under the data-dependent optimal scaling, the learning dynamics is independent of data and weakly dependent on depth, resulting in a constant linear convergence rate across all depths including infinity. We further show similar data-dependent effects in deep scalar linear networks with residual connections.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。