用范数约束的线性优化器加速深度学习训练，无需Adam也能快。
Training Deep Learning Models with Norm-Constrained LMOs
- 基于范数球上的线性优化器设计新算法，自适应问题几何结构。
- 在nanoGPT上实现显著提速，且超参可在不同模型尺寸间复用。
- 内存高效，仅需存储一组权重和梯度，支持半精度存储。
本文研究利用范数球上线性最小化算子（LMO）的优化方法。提出一种新的随机算法族，通过LMO适应问题几何特性，出人意料地可应用于无约束问题。该方法统一了多种现有优化算法。进一步提出适用于深度架构的显式范数选择，带来超参数跨模型尺寸的可转移性。实验表明，在nanoGPT训练中使用所提算法Scion可获得显著加速，且不依赖Adam。该方法内存高效，仅需存储一组模型权重和一组梯度，支持半精度存储。代码已开源：https://github.com/LIONS-EPFL/scion。
原文摘要 · Abstract (English)
In this work, we study optimization methods that leverage the linear minimization oracle (LMO) over a norm-ball. We propose a new stochastic family of algorithms that uses the LMO to adapt to the geometry of the problem and, perhaps surprisingly, show that they can be applied to unconstrained problems. The resulting update rule unifies several existing optimization methods under a single framework. Furthermore, we propose an explicit choice of norm for deep architectures, which, as a side benefit, leads to the transferability of hyperparameters across model sizes. Experimentally, we demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam. The proposed method is memory-efficient, requiring only one set of model weights and one set of gradients, which can be stored in half-precision. The code is available at https://github.com/LIONS-EPFL/scion .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。