无需计算完整梯度，通过打乱数据提升随机优化效率
Variance Reduction Methods Do Not Need to Compute Full Gradients: Improved Efficiency through Shuffling
- 用数据打乱代替定期计算全梯度，降低计算开销
- 非凸目标下收敛速度媲美传统打乱方法，强凸时更优
- 适合大规模机器学习任务，已在CIFAR-10/100验证
随机优化算法广泛用于大规模机器学习，但其收敛常受非消失方差影响。方差减少（VR）方法如SVRG和SARAH虽能缓解此问题，却因需周期性计算完整梯度而成为瓶颈。本文探索主流VR技术，提出一种无需昂贵全梯度计算的新方法。通过引入打乱启发式与SAG/SAGA思想，实现内存高效。在非凸目标下，收敛率与标准打乱方法相当；强凸条件下则表现更优。实验验证了该方法在图像分类任务（CIFAR-10和CIFAR-100）上的高效性与可扩展性。
原文摘要 · Abstract (English)
Stochastic optimization algorithms are widely used for machine learning with large-scale data. However, their convergence often suffers from non-vanishing variance. Variance Reduction (VR) methods, such as SVRG and SARAH, address this issue but introduce a bottleneck by requiring periodic full gradient computations. In this paper, we explore popular VR techniques and propose an approach that eliminates the necessity for expensive full gradient calculations. To avoid these computations and make our approach memory-efficient, we employ two key techniques: the shuffling heuristic and the concept of SAG/SAGA methods. For non-convex objectives, our convergence rates match those of standard shuffling methods, while under strong convexity, they demonstrate an improvement. We empirically validate the efficiency of our approach and demonstrate its scalability on large-scale machine learning tasks including image classification problem on CIFAR-10 and CIFAR-100 datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。