增大批量大小可加速黎曼随机梯度下降收敛，且降低计算开销。
Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size
- 采用递增批量大小替代固定批量，提升优化效率。
- 收敛速率从O(T⁻¹+C)提升至O(T⁻¹)，T为迭代次数。
- 兼顾小批量的低方差与大批量的高效率，适合大规模优化场景。
我们理论分析了黎曼随机梯度下降（RSGD）的收敛行为，发现使用递增批量大小相比固定批量大小能实现更快收敛，无论学习率恒定还是衰减（如余弦退火和多项式衰减）。收敛速率由固定批量下的O(T⁻¹+C)提升至递增批量下的O(T⁻¹)，其中T为总迭代次数，C为常数。通过主成分分析和低秩矩阵补全，我们从理论和数值上研究了递增批量对计算时间的影响，以随机一阶预言机（SFO）复杂度衡量。结果表明，递增批量可显著降低RSGD的SFO复杂度。此外，递增批量兼具小批量的低方差优势与大批量的高效率，优于单一固定批量策略。
原文摘要 · Abstract (English)
We theoretically analyzed the convergence behavior of Riemannian stochastic gradient descent (RSGD) and found that using an increasing batch size leads to faster convergence than using a constant batch size, not only with a constant learning rate but also with a decaying learning rate, such as cosine annealing decay and polynomial decay. The convergence rate improves from $O(T^{-1}+C)$ with a constant batch size to $O(T^{-1})$ with an increasing batch size, where $T$ denotes the total number of iterations and $C$ is a constant. Using principal component analysis and low-rank matrix completion, we investigated, both theoretically and numerically, how an increasing batch size affects computational time as quantified by stochastic first-order oracle (SFO) complexity. An increasing batch size was found to reduce the SFO complexity of RSGD. Furthermore, an increasing batch size was found to offer the advantages of both small and large constant batch sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。