用信任域框架统一解释Adam等优化器的动量与学习率机制。
A Trust-region Framework for Moment Estimation
- 基于高阶矩的信赖域约束，推导出自适应学习率方法。
- 四阶矩在弱约束下效果最好，二阶矩在强约束下更优。
- 为优化器设计提供理论统一视角，适合研究者参考。
本文提出一种信任域框架，用于理解随机梯度优化中自适应动量估计机制（如Adam）的行为。具体而言，每个参数的更新步长受有限阶p-阶矩信任域约束，其中p≥1。由此推导出一类基于二阶矩估计和归一化p-阶矩估计的学习率机制。当p=4时，涉及峰度估计。后续推导揭示了基于矩估计的归一化、学习率调度、动量作为谱级一阶低通正则化，以及算子级谱范数归一化的统一解释，均在共同的信任域框架下成立。在GPT2-124M于FineWeb-Edu和TinyStories上的初步实验表明，当信任域约束较弱时，四阶矩实现表现最佳；随着约束逐渐增强，二阶矩实现变得更具竞争力，通常取得略低于对应四阶矩实现的验证损失。
原文摘要 · Abstract (English)
In this paper, we develop a trust-region framework for understanding the behavior of adaptive moment estimation mechanisms, such as \textsc{Adam}, in stochastic gradient optimization. Specifically, the magnitude of the update step associated with each individual parameter is constrained by a finite-order $p$-moment trust-region, with $p\ge1$. The resulting derivation leads to a family of learning-rate mechanisms based on second-moment estimation and normalized $p$-th-moment estimation. For $p=4$, this involves kurtosis estimation. Subsequent derivations provide a unified interpretation of moment-estimation-based normalization, learning-rate scheduling, momentum as a spectral first-order lowpass regularization, and operator-level spectral-norm normalization within a common trust-region framework. Preliminary experiments on GPT2-124M trained on FineWeb-Edu and TinyStories suggest that the fourth-moment realization provides its greatest benefit when trust-region constraints are weak. As progressively stronger trust-region controls are introduced, the second-moment realization becomes increasingly competitive, often achieving slightly lower validation loss than its corresponding fourth-moment realization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。