用非线性预条件信任域法实现并行训练，省去调参且精度不降。
Data-Parallel Neural Network Training via Nonlinearly Preconditioned Trust-Region Method
- 基于数据并行构造非线性预条件器，动态调整迭代步长。
- 在MNIST和CIFAR-10上达到与SGD、Adam相当的验证准确率。
- 无需昂贵超参数调优，适合大规模分布式训练场景。
随着模型和数据规模持续增长，并行训练方法在机器学习中日益重要。本文提出一种改进的加性预条件信任域策略（APTS）用于深度神经网络训练。该方法采用数据并行方式构建非线性预条件器，融入非线性优化框架。与常用梯度下降变体如随机梯度下降（SGD）和自适应矩估计（Adam）不同，该方法在每轮迭代中隐式调整步长，从而避免了高昂的超参数调优成本。我们在MNIST和CIFAR-10数据集上验证了所提APTS变体的性能，结果表明其在保持并行训练能力的同时，达到了与SGD和Adam相当的验证准确率，显著降低了调参负担。
原文摘要 · Abstract (English)
Parallel training methods are increasingly relevant in machine learning (ML) due to the continuing growth in model and dataset sizes. We propose a variant of the Additively Preconditioned Trust-Region Strategy (APTS) for training deep neural networks (DNNs). The proposed APTS method utilizes a data-parallel approach to construct a nonlinear preconditioner employed in the nonlinear optimization strategy. In contrast to the common employment of Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam), which are both variants of gradient descent (GD) algorithms, the APTS method implicitly adjusts the step sizes in each iteration, thereby removing the need for costly hyperparameter tuning. We demonstrate the performance of the proposed APTS variant using the MNIST and CIFAR-10 datasets. The results obtained indicate that the APTS variant proposed here achieves comparable validation accuracy to SGD and Adam, all while allowing for parallel training and obviating the need for expensive hyperparameter tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。