通过利用LLM对称性，动态优化量化基底，显著降低困惑度。
GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
- 在训练中引入LogSumExp项打破对称性，选择最优量化基底。
- W4A4下困惑度从8.22降至6.73，优于需校准数据的后训练方法。
- 无需校准数据或模拟量化，训练开销极小，适合部署场景。
Transformer模型具有使输出保持不变但改变量化的内部连续对称性。GaugeQuant通过在损失函数中引入LogSumExp项来打破这些对称性，从而选择能最小化激活异常值的基底。通过使用停止梯度操作，仅更新旋转矩阵，确保语言建模目标完全不变。该方法无需特定校准数据、无需量化模拟，且训练开销可忽略不计。在LLaMA-2 7B模型上，采用每组128个元素的分组量化（W4A4）时，困惑度从8.22降至6.73；在W4A16下，困惑度从11.16降至5.45，性能媲美需冻结模型和校准数据的后训练方法。代码已公开于https://github.com/MPedraBento/gauge-quant。
原文摘要 · Abstract (English)
Transformers are known to have internal continuous symmetries that leave outputs invariant, while modifying quantization. GaugeQuant leverages this in-training by introducing a LogSumExp term to the loss that breaks the symmetries, thus selecting a basis that minimizes activation outliers. A stop-gradient operator ensures that only rotation matrices are updated, yielding the language modeling objective completely unaltered. Our requires no specific calibration data, no quantization simulation, and adds negligible training overhead. With the LLaMA-2 7B model under W4A4 quantization with group size 128, perplexity drops from 8.22 to 6.73, competing with post-training methods that require frozen models and calibration datasets. Under W4A16, perplexity drops from 11.16 to 5.45. Code is available at https://github.com/MPedraBento/gauge-quant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。