分析Lion优化器在集中与分布式场景下的收敛性，提出改进方案提升训练效率。
Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings
- 基于标准假设，推导出Lion的收敛速率为O(d^{1/2}T^{-1/4})
- 引入方差减少后，收敛率提升至O(d^{1/2}T^{-1/3})
- 设计通信高效变体，支持双向符号压缩，适合大规模分布式训练
本文分析了Lion优化器的收敛性质。首先，在标准假设下，证明其收敛速率为$\/mathcal{O}(d^{1/2}T^{-1/4})$，其中$d$为问题维度，$T$为迭代次数。为进一步提升速率，提出带方差减少的Lion优化器，收敛率提升至$\/mathcal{O}(d^{1/2}T^{-1/3})$。在分布式设置中，标准与方差减少版本分别达到$\/mathcal{O}(d^{1/2}(nT)^{-1/4})$和$\/mathcal{O}(d^{1/2}(nT)^{-1/3})$的收敛率，其中$n$为节点数。此外，研究了一种通信高效的分布式Lion变体，实现双向符号压缩。通过无偏符号操作，该变体及其方差减少版本分别达到$\/mathcal{O}\\left( \max \\{\frac{d^{1/4}}{T^{1/4}}, \frac{d^{1/10}}{n^{1/5}T^{1/5}} \\} \\right)$和$\/mathcal{O}\\left( \frac{d^{1/4}}{T^{1/4}} \\right)$的收敛率。
原文摘要 · Abstract (English)
In this paper, we analyze the convergence properties of the Lion optimizer. First, we establish that the Lion optimizer attains a convergence rate of $\mathcal{O}(d^{1/2}T^{-1/4})$ under standard assumptions, where $d$ denotes the problem dimension and $T$ is the iteration number. To further improve this rate, we introduce the Lion optimizer with variance reduction, resulting in an enhanced convergence rate of $\mathcal{O}(d^{1/2}T^{-1/3})$. We then analyze in distributed settings, where the standard and variance reduced version of the distributed Lion can obtain the convergence rates of $\mathcal{O}(d^{1/2}(nT)^{-1/4})$ and $\mathcal{O}(d^{1/2}(nT)^{-1/3})$, with $n$ denoting the number of nodes. Furthermore, we investigate a communication-efficient variant of the distributed Lion that ensures sign compression in both communication directions. By employing the unbiased sign operations, the proposed Lion variant and its variance reduction counterpart, achieve convergence rates of $\mathcal{O}\left( \max \left\{\frac{d^{1/4}}{T^{1/4}}, \frac{d^{1/10}}{n^{1/5}T^{1/5}} \right\} \right)$ and $\mathcal{O}\left( \frac{d^{1/4}}{T^{1/4}} \right)$, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。