提出大批次联邦学习新方法，兼顾训练速度与模型精度。
On Using Large-Batches in Federated Learning
- 结合小批量与大批量优势，优化联邦学习中的批处理策略。
- 在相同迭代次数下，ResNet50和VGG11测试准确率分别提升32.33%和3.74%。
- 适合资源受限但需高效训练的设备端联邦学习场景。
高效的联邦学习（FL）对于在计算资源有限且网络带宽受限的设备上训练深度网络至关重要。随着大数据的发展，设备需生成或收集多模态数据以训练通用或本地上下文感知的模型，尤其在数据隐私和本地性至关重要的场景下。现有FL算法通常在并行性能与统计性能之间权衡：提高模型质量往往需要更高的通信频率，反之亦然。在频繁同步设置下，通过增大全局批次大小可显著提升每轮训练的并行效率，从而实现可观的训练加速。然而，这可能导致测试性能下降（如测试损失或准确率降低），源于大批次训练带来的泛化能力退化问题。为解决该问题，本文提出一种融合小批量与大批量训练优势的新思路，探索同时实现大规模并行扩展与良好泛化能力的新方向。在相同迭代次数下，所提方法在ResNet50和VGG11模型上的测试准确率分别较传统小批量训练提升32.33%和3.74%。
原文摘要 · Abstract (English)
Efficient Federated learning (FL) is crucial for training deep networks over devices with limited compute resources and bounded networks. With the advent of big data, devices either generate or collect multimodal data to train either generic or local-context aware networks, particularly when data privacy and locality is vital. FL algorithms generally trade-off between parallel and statistical performance, improving model quality at the cost of higher communication frequency, or vice versa. Under frequent synchronization settings, FL over a large cluster of devices may perform more work per-training iteration by processing a larger global batch-size, thus attaining considerable training speedup. However, this may result in poor test performance (i.e., low test loss or accuracy) due to generalization degradation issues associated with large-batch training. To address these challenges with large-batches, this work proposes our vision of exploiting the trade-offs between small and large-batch training, and explore new directions to enjoy both the parallel scaling of large-batches and good generalizability of small-batch training. For the same number of iterations, we observe that our proposed large-batch training technique attains about 32.33% and 3.74% higher test accuracy than small-batch training in ResNet50 and VGG11 models respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。