用可训练优化器替代传统方法，加速模型训练且效果更优。
A Trainable Optimizer
- 设计可联合训练的优化策略，用线性近似替代梯度估计
- 在强凸与非凸场景中收敛速度超越ADAM等基准算法
- 计算开销极低，适合大模型微调等实际应用
学习优化的核心思想是使用可训练的优化策略，而非依赖手动定义的完整梯度估计（如ADAM）。本文提出一个框架，联合训练完整的梯度估计器和模型的可训练参数。具体而言，我们证明了伪线性可训练优化器（Pseudo-linear TO）——一种完整梯度的线性近似——在保持与SGD相同收敛速率的同时，有效降低方差。伪线性TO仅需额外少量张量乘法，计算开销可忽略不计。为进一步提升效率，我们引入两种简化变体。实验表明，该类优化方法在强凸与非凸设置下均比基准算法（如ADAM）收敛更快，并在大语言模型微调任务中表现优异。
原文摘要 · Abstract (English)
The concept of learning to optimize involves utilizing a trainable optimization strategy rather than relying on manually defined full gradient estimations such as ADAM. We present a framework that jointly trains the full gradient estimator and the trainable weights of the model. Specifically, we prove that pseudo-linear TO (Trainable Optimizer), a linear approximation of the full gradient, matches SGD's convergence rate while effectively reducing variance. Pseudo-linear TO incurs negligible computational overhead, requiring only minimal additional tensor multiplications. To further improve computational efficiency, we introduce two simplified variants of Pseudo-linear TO. Experiments demonstrate that TO methods converge faster than benchmark algorithms (e.g., ADAM) in both strongly convex and non-convex settings, and fine tuning of an LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。