AdaHOP动态调整低精度训练中的哈达玛变换，提升模型质量与速度。
AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation
- 根据数据异常模式自适应选择哈达玛变换或异常提取策略。
- 在MXFP4下实现接近BF16的精度，内存压缩达3.6倍。
- 适合追求高效低精度大模型训练的研究者与工程师。
哈达玛变换已成为稳定低精度训练的关键工具，但现有方法对张量和计算路径采用统一应用策略。我们发现这种‘一刀切’方式存在固有局限：只有当哈达玛平滑方向与操作数的异常结构对齐时，才能有效降低量化误差。通过对大语言模型训练中权重、激活值和梯度的系统研究，我们识别出三种稳定的异常模式——行式、列式和无异常，并证明矩阵乘法中每种异常模式组合需采用不同的变换或异常处理策略。为此提出AdaHOP（自适应哈达玛变换，异常模式感知），在内维混合能有效抑制异常时使用内层哈达玛变换（IHT），否则选择性地将主导的异常行或列提取至高精度路径（OE）。结合硬件感知的融合Triton内核，AdaHOP可在MXFP4精度下从头训练，达到BF16级性能，同时实现最高3.6倍内存压缩与1.46倍端到端训练加速。
原文摘要 · Abstract (English)
Hadamard transforms have become a key tool for stabilizing low-precision training, but existing methods apply them uniformly across tensors and computation paths. We show that this one-size-fits-all strategy is inherently limited: Hadamard smoothing reduces quantization error only when its direction is properly aligned with the operand's outlier structure. Through a systematic study of weights, activations, and gradients in LLM training, we identify three stable outlier patterns, Row-wise, Column-wise, and None, and show that each outlier pattern pair in matrix multiplication requires a distinct transform or outlier-handling strategy. We propose AdaHOP, Adaptive Hadamard transform with Outlier-Pattern-aware strategy, which applies Inner Hadamard Transform (IHT) when inner-dimension mixing properly suppresses the operands' outliers, and selectively applies Outlier Extraction (OE) that extracts dominant outlier rows or columns into a high-precision path when it does not. With fused, hardware-aware Triton kernels, AdaHOP enables training from scratch at MXFP4 precision with BF16-level quality, while achieving up to 3.6X memory compression, 1.46X end-to-end training speedup over BF16.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。