用凸优化方法训练大规模神经网络，突破了传统方法的规模限制。
CRONOS: Enhancing Deep Learning with Scalable GPU Accelerated Convex Neural Networks
- 基于凸重构技术，设计可扩展的优化算法CRONOS
- 在ImageNet等大数据集上实现全局最优解，精度媲美主流优化器
- 适用于多层网络，适合需要理论保障的高性能场景
我们提出CRONOS算法，用于两层神经网络的凸优化。CRONOS是首个能扩展到高维数据集（如ImageNet）的算法，显著超越以往仅限于下采样后的MNIST和CIFAR-10的研究。以CRONOS为基础，进一步开发了结合交替最小化的CRONOS-AM算法，可训练任意架构的多层网络。理论分析证明，在弱假设下CRONOS可收敛至凸重构的全局最小值。通过JAX中大规模GPU加速实验验证，CRONOS-AM在ImageNet和IMDb等基准数据集上的验证精度可媲美甚至优于主流深度学习优化器。据我们所知，CRONOS是首个利用凸重构提升大规模学习任务性能的算法。
原文摘要 · Abstract (English)
We introduce the CRONOS algorithm for convex optimization of two-layer neural networks. CRONOS is the first algorithm capable of scaling to high-dimensional datasets such as ImageNet, which are ubiquitous in modern deep learning. This significantly improves upon prior work, which has been restricted to downsampled versions of MNIST and CIFAR-10. Taking CRONOS as a primitive, we then develop a new algorithm called CRONOS-AM, which combines CRONOS with alternating minimization, to obtain an algorithm capable of training multi-layer networks with arbitrary architectures. Our theoretical analysis proves that CRONOS converges to the global minimum of the convex reformulation under mild assumptions. In addition, we validate the efficacy of CRONOS and CRONOS-AM through extensive large-scale numerical experiments with GPU acceleration in JAX. Our results show that CRONOS-AM can obtain comparable or better validation accuracy than predominant tuned deep learning optimizers on vision and language tasks with benchmark datasets such as ImageNet and IMDb. To the best of our knowledge, CRONOS is the first algorithm which utilizes the convex reformulation to enhance performance on large-scale learning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。