HOT通过哈达玛优化，显著降低训练内存与计算开销。
HOT: Hadamard-based Optimized Training
- 采用哈达玛量化与低秩近似，按路径选择性优化反向传播
- 实测内存减少75%，速度提升2.6倍，精度损失极小
- 适合追求高效训练的模型部署与资源受限场景
随着训练成本不断上升，优化反向传播以减少内存占用和计算开销变得愈发重要。然而，这一目标面临多重约束，需兼顾多种指标并保持训练质量。本文聚焦于占训练成本最大比例的矩阵乘法，深入分析其反向传播过程，识别出轻量级但高效的优化技术。基于此，提出一种新方法——哈达玛优化训练（HOT）。该方法在不同反向路径中选择性应用哈达玛量化与哈达玛低秩近似，并引入激活缓冲区压缩与分层量化器选择两项增强策略。实验表明，HOT在真实GPU上实现最高75%的内存节省和2.6倍加速，相较于FP32精度仅产生可忽略的精度损失。
原文摘要 · Abstract (English)
It has become increasingly important to optimize backpropagation to reduce memory usage and computational overhead. Achieving this goal is highly challenging, as multiple objectives must be considered jointly while maintaining training quality. In this paper, we focus on matrix multiplication, which accounts for the largest portion of training costs, and analyze its backpropagation in detail to identify lightweight techniques that offer the best benefits. Based on this analysis, we introduce a novel method, Hadamard-based Optimized Training (HOT). In this approach, we apply Hadamard-based optimizations, such as Hadamard quantization and Hadamard low-rank approximation, selectively and with awareness of the suitability of each optimization for different backward paths. Additionally, we introduce two enhancements: activation buffer compression and layer-wise quantizer selection. Our extensive analysis shows that HOT achieves up to 75% memory savings and a 2.6 times acceleration on real GPUs, with negligible accuracy loss compared to FP32 precision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。