研究随机动量方法在计算效率与运行时间间的权衡,揭示大批次的优势边界。
Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods
- 分析重球法与加速SGD在不同谱分布下的批量大小权衡机制。
- 发现重球法在更大批次窗口内保持SGD级计算效率,窗口可扩大√κ倍。
- 加速SGD在小批量时更高效,但随批量增大转向更快收敛,适合不同场景。
随机动量方法如重球法(HB)、Nesterov动量及加速SGD(ASGD)广泛用于现代训练,其随机优势取决于串行运行时间(达到目标精度所需迭代次数)和计算效率(反向梯度查询或浮点操作总成本)。当收缩间隙随批量大小线性增长时,大批次可降低串行运行时间而不损害计算效率。本文针对高斯协方差的线性回归问题,研究随机HB与ASGD,并建立有限维、离散时间下的批大小权衡下界。结果表明:重球法不优于SGD的计算效率前沿,但在更宽的批量窗口内保持其效率水平,允许更大批次降低串行运行时间,直至达到确定性加速尺度;该窗口可比SGD临界批量大√κ倍。对于加速SGD,表现依赖于谱特性:对快速衰减的幂律谱,小批量时优于HB/SGD的计算效率;但随着批量增大,以牺牲计算效率为代价换取更好的串行运行时间。合成线性回归实验验证了这些定性规律,包括慢衰减谱下ASGD与HB近似重叠,以及快速衰减谱下的预测计算效率-运行时间权衡。
原文摘要 · Abstract (English)
Stochastic momentum methods such as heavy ball (HB), Nesterov momentum, and variants of Accelerated SGD (ASGD) [Kidambi et al., 2018] are widely used in modern training, but their stochastic benefits depend on two distinct quantities: serial runtime, the number of iterations needed to reach a target accuracy, and compute efficiency (CE), the inverse total gradient-query or FLOP cost. Larger batches reduce serial runtime without hurting CE only when the contraction gap grows linearly with batch size. We study stochastic HB and ASGD for consistent linear regression with Gaussian covariates and prove finite-dimensional, discrete-time lower bounds on their batch-size tradeoffs. Our first result shows that HB does not improve the CE frontier over SGD for arbitrary spectra; rather, it preserves SGD-level CE over a larger batch-size window, allowing larger batches to reduce serial runtime until HB reaches its deterministic accelerated scale. This window can be a factor $\sqrtκ$ larger than the SGD critical batch size. For ASGD, the picture is more spectrum-dependent: for rapidly decaying power-law spectra, ASGD improves small-batch CE over HB/SGD, but as batch size grows it trades this CE advantage for improved serial runtime. Synthetic linear-regression experiments verify these qualitative regimes, including near-overlap of ASGD and HB for slowly decaying spectra and the predicted CE--serial tradeoff for rapidly decaying spectra.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。