根据梯度多样性动态调整批量大小，加速模型训练。
DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation
- 基于梯度多样性自动调节批量大小，平衡效率与泛化能力。
- 在CIFAR-10/100和Tiny-ImageNet上比标准SGD快1.06至5.0倍。
- 适合追求训练速度且可接受轻微性能损失的深度学习研究者。
本文旨在加速机器学习模型的训练，这对大规模深度神经网络而言至关重要。虽然随机梯度下降（SGD）及其变体被广泛用于训练深度神经网络，但传统方法多聚焦于学习率调优。本文提出一种新型自适应批量大小的SGD算法——DiveBatch，可动态调整批量大小。增大批量虽能提升并行计算效率，但小批量训练通常收敛更快且泛化性能更优。为解决这一矛盾，DiveBatch基于梯度多样性进行数据驱动的自适应，使模型在保持小批量泛化性能的同时，提升收敛速度与计算效率。梯度多样性具有坚实的理论基础，源自SGD的收敛分析。在合成数据及CIFAR-10、CIFAR-100、Tiny-ImageNet上的实验表明，DiveBatch相比标准SGD和AdaBatch（1.06–5.0倍）显著加速收敛，性能略有下降。
原文摘要 · Abstract (English)
The goal of this paper is to accelerate the training of machine learning models, a critical challenge since the training of large-scale deep neural models can be computationally expensive. Stochastic gradient descent (SGD) and its variants are widely used to train deep neural networks. In contrast to traditional approaches that focus on tuning the learning rate, we propose a novel adaptive batch size SGD algorithm, DiveBatch, that dynamically adjusts the batch size. Adapting the batch size is challenging: using large batch sizes is more efficient due to parallel computation, but small-batch training often converges in fewer epochs and generalizes better. To address this challenge, we introduce a data-driven adaptation based on gradient diversity, enabling DiveBatch to maintain the generalization performance of small-batch training while improving convergence speed and computational efficiency. Gradient diversity has a strong theoretical justification: it emerges from the convergence analysis of SGD. Evaluations of DiveBatch on synthetic and CiFar-10, CiFar-100, and Tiny-ImageNet demonstrate that DiveBatch converges significantly faster than standard SGD and AdaBatch (1.06 -- 5.0x), with a slight trade-off in performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。