针对神经网络各层差异，提出自适应分层量化方法，提升分布式训练效率。
Layer-wise Quantization for Quantized Optimistic Dual Averaging
- 按层分别量化，动态适应不同层的结构与表征特性
- 在12块以上GPU上训练Wasserstein GAN时提速最高达150%
- 适合需要高效分布式训练的大规模模型场景
现代深度神经网络因结构差异（如维度、激活函数等）在不同层间呈现异质性，影响预测效果。本文提出一种通用的分层量化框架，具备紧致的方差与码长界，可随训练过程自适应调整。进一步将该技术应用于分布式变分不等式中，提出新型量化乐观对偶平均（QODA）算法，支持自适应学习率，在单调变分不等式问题中实现有竞争力的收敛速度。实验表明，QODA在12块以上GPU上训练Wasserstein GAN时，端到端训练时间相比基线最多提升150%。
原文摘要 · Abstract (English)
Modern deep neural networks exhibit heterogeneity across numerous layers of various types such as residuals, multi-head attention, etc., due to varying structures (dimensions, activation functions, etc.), distinct representation characteristics, which impact predictions. We develop a general layer-wise quantization framework with tight variance and code-length bounds, adapting to the heterogeneities over the course of training. We then apply a new layer-wise quantization technique within distributed variational inequalities (VIs), proposing a novel Quantized Optimistic Dual Averaging (QODA) algorithm with adaptive learning rates, which achieves competitive convergence rates for monotone VIs. We empirically show that QODA achieves up to a $150\%$ speedup over the baselines in end-to-end training time for training Wasserstein GAN on $12+$ GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。