提出轻量级量化训练框架SASQ，实现高精度静态量化推理。
SASQ: Static Activation Scaling for Quantization-Aware Training in Large Language Models
- 仅优化激活量化因子，不修改预训练权重
- 在LLaMA2-7B上比QuaRot低5.2%困惑度
- 适合边缘设备部署，兼顾精度与效率
大型语言模型在自然语言任务中表现优异，但其规模增长远超GPU内存发展，带来部署挑战。模型量化通过降低权重和激活精度缓解此问题，但现有方案存在根本权衡：动态量化计算开销高，部署困难；静态量化牺牲精度。现有量化感知训练（QAT）方法还面临权重训练成本高的问题。本文提出SASQ：一种专为激活量化因子设计的轻量级QAT框架。SASQ仅优化量化因子（不更新预训练权重），实现静态推理下的高精度与高效部署。通过自适应截断部分异常值，降低量化难度的同时保持激活分布特性。SASQ不仅超越现有最先进量化方案，还优于对应FP16模型。在LLaMA2-7B上，其在WikiText2上的困惑度比QuaRot低5.2%，比FP16模型低4.7%。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at natural language tasks but face deployment challenges due to their growing size outpacing GPU memory advancements. Model quantization mitigates this issue by lowering weight and activation precision, but existing solutions face fundamental trade-offs: dynamic quantization incurs high computational overhead and poses deployment challenges on edge devices, while static quantization sacrifices accuracy. Existing approaches of quantization-aware training (QAT) further suffer from weight training costs. We propose SASQ: a lightweight QAT framework specifically tailored for activation quantization factors. SASQ exclusively optimizes only the quantization factors (without changing pre-trained weights), enabling static inference with high accuracy while maintaining deployment efficiency. SASQ adaptively truncates some outliers, thereby reducing the difficulty of quantization while preserving the distributional characteristics of the activations. SASQ not only surpasses existing SOTA quantization schemes but also outperforms the corresponding FP16 models. On LLaMA2-7B, it achieves 5.2% lower perplexity than QuaRot and 4.7% lower perplexity than the FP16 model on WikiText2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。