揭示小批量梯度下降的隐式偏置,解析批量大小与动量的影响。
The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradient
- 基于不同范数下的最速下降框架,统一分析优化行为。
- 无动量时仅全批量能保证收敛与分类成功,动量可实现近似最大间隔解。
- 首次给出无需维度依赖的显式收敛率,适合研究优化器设计者。
多种常用优化方法(如SignSGD、Muon)可视为在不同范数诱导几何下的最速下降实例。本文研究多分类任务中小批量随机最速下降的隐式偏置,揭示批量大小、动量和方差缩减如何影响极限最大间隔行为与收敛速率,适用于逐元素及Schatten-p范数。结果表明:无动量时,仅全批量梯度可保证最坏情况收敛与正确分类;而引入动量后,通过批大小与动量的权衡,小批量也能收敛至近似最大间隔解,尽管收敛变慢。该方法提供完全显式的、维度无关的收敛速率,优于以往结果。此外,证明方差缩减可在任意批量下恢复精确的全批量隐式偏置,但收敛更慢。最后,对单样本最速下降(无动量)进行深入分析,通过具体数据例证其收敛至根本不同的偏置,揭示纯随机更新的关键局限。本工作统一解析了随机优化何时与全批量行为一致,为深入理解随机梯度最速下降算法的训练行为奠定基础。
原文摘要 · Abstract (English)
A variety of widely used optimization methods like SignSGD and Muon can be interpreted as instances of steepest descent under different norm-induced geometries. In this work, we study the implicit bias of mini-batch stochastic steepest descent in multi-class classification, characterizing how batch size, momentum, and variance reduction shape the limiting max-margin behavior and convergence rates under general entry-wise and Schatten-$p$ norms. We show that, without momentum, worst-case convergence and successful classification can only be guaranteed with full-batch gradient. In contrast, momentum enables small-batch convergence to an approximate max-margin solution through a batch-momentum trade-off, though it slows convergence. This approach provides fully explicit, dimension-free rates that improve upon prior results. Moreover, we prove that variance reduction can recover the exact full-batch implicit bias for any batch size, albeit at a slower convergence rate. Finally, we further investigate the batch-size-one steepest descent without momentum, and reveal its convergence to a fundamentally different bias via a concrete data example, which reveals a key limitation of purely stochastic updates. Overall, our unified analysis clarifies when stochastic optimization aligns with full-batch behavior, and paves the way for perform deeper explorations of the training behavior of stochastic gradient steepest descent algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。