为未来硬件设计提出适配位宽的对数定点训练方法，显著降低计算开销。
Bitwidth-Specific Logarithmic Arithmetic for Future Hardware-Accelerated Training
- 根据不同位宽优化对数加法的分段线性近似
- 12位整数训练在CIFAR-100和TinyImageNet上精度损失极小
- 硬件测试显示面积减少32.5%，功耗降低53.5%
尽管量化技术已大幅降低深度学习推理的计算成本，但训练仍主要依赖复杂的浮点运算。低精度定点训练提供了一种有吸引力的替代方案。本文针对未来硬件加速器设计，提出一种新颖的低精度对数定点训练增强方法，将位宽纳入算术运算近似的设计考量。我们引入一种新型硬件友好的分段线性近似方法用于对数加法，并通过模拟退火在不同精度级别下进行优化。使用C++位真仿真，在CIFAR-100和TinyImageNet上分别实现了VGG-11和VGG-16模型的12位整数训练，与32位浮点训练相比精度下降极小。硬件评估表明，所提出的对数定点乘累加单元相较线性定点方案，面积最多减少32.5%，能耗降低53.5%。
原文摘要 · Abstract (English)
While advancements in quantization have significantly reduced the computational costs of inference in deep learning, training still predominantly relies on complex floating-point arithmetic. Low-precision fixed-point training presents a compelling alternative. This work introduces a novel enhancement in low-precision logarithmic fixed-point training, geared towards future hardware accelerator designs. We propose incorporating bitwidth in the design of approximations to arithmetic operations. To this end, we introduce a new hardware-friendly, piece-wise linear approximation for logarithmic addition. Using simulated annealing, we optimize this approximation at different precision levels. A C++ bit-true simulation demonstrates training of VGG-11 and VGG-16 models on CIFAR-100 and TinyImageNet, respectively, using 12-bit integer arithmetic with minimal accuracy degradation compared to 32-bit floating-point training. Our hardware study reveals up to 32.5% reduction in area and 53.5% reduction in energy consumption for the proposed LNS multiply-accumulate units compared to that of linear fixed-point equivalents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。