揭示优化器记忆如何隐式改变损失函数，解释不同模型泛化差异
How Memory in Optimization Algorithms Implicitly Modifies the Loss
- 用当前参数替换历史参数，再加修正项，构建无记忆近似算法
- 发现Lion缺乏AdamW那种由记忆引发的隐式抗正则化效应
- 为Lion更好的泛化性能提供了理论解释，适合研究优化器机制者阅读
在深度学习中，现代优化方法的每次更新都依赖于历史迭代信息，即所谓的‘记忆’，且这种依赖随时间快速衰减。例如，带动量的梯度下降通过指数平均过去梯度实现指数衰减的记忆。本文提出一种通用技术：将更新公式中的所有历史参数替换为当前参数，并添加一个由记忆引起的修正项。该修正项可被理解为对损失函数的扰动，其性质揭示了记忆如何隐式（反）正则化优化过程。作为应用，我们发现Lion并未像AdamW那样产生由记忆诱导的隐式抗正则化，这为近期文献中记录的Lion更优泛化性能提供了理论解释。
原文摘要 · Abstract (English)
In modern optimization methods used in deep learning, each update depends on the history of previous iterations, often referred to as memory, and this dependence decays fast as the iterates go further into the past. For example, gradient descent with momentum has exponentially decaying memory through exponentially averaged past gradients. We introduce a general technique for identifying a memoryless algorithm that approximates an optimization algorithm with memory. It is obtained by replacing all past iterates in the update by the current one, and then adding a correction term arising from memory (also a function of the current iterate). This correction term can be interpreted as a perturbation of the loss, and the nature of this perturbation can inform how memory implicitly (anti-)regularizes the optimization dynamics. As an application of our theory, we find that Lion does not have the kind of implicit anti-regularization induced by memory that AdamW does, providing a theory-based explanation for Lion's better generalization performance recently documented.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。