用随机舍入训练低比特大模型，小批量也能收敛。
Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding
- 用随机舍入减少训练中的精度损失，保持梯度无偏。
- 增大批次可补偿低精度带来的误差，提升收敛性。
- 适合资源受限的边缘设备部署大模型训练。
大模型训练成本高昂。量化训练虽能提升计算与内存效率，但引入量化噪声，影响收敛并降低精度。随机舍入（SR）理论上可提供无偏梯度估计，是确定性舍入的更优替代方案。然而，其与训练中其他因素（尤其是批次大小）的交互机制尚未充分研究。本文从理论与实证两方面分析了使用随机舍入的最小批量随机梯度下降（SGD），发现增大批次大小可有效补偿反向传播中的精度降低。此外，权重与激活量化的梯度方差影响机制不同。实验验证了这些理论发现。
原文摘要 · Abstract (English)
LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased gradient estimates. However, its interaction with other training factors -- especially batch size -- remains under explored. In this paper, we present a theoretical and empirical study of mini-batch stochastic gradient descent (SGD) with SR, showing that increased batch sizes can compensate for reduced precision during back-propagation. Furthermore, we show that quantizing weights and activations impacts gradient variance in distinct ways. Our experiments validate these theoretical insights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。