让大模型量化更快更省资源，单张显卡搞定70B模型旋转校准
DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
- 通过约束激活分布，简化旋转优化复杂度
- 70B模型旋转优化提速47倍、内存减少10倍
- 首次单3090显卡完成70B模型旋转校准，适合资源受限场景
量化对加速大规模模型推理至关重要，旋转矩阵能通过平滑异常值提升量化性能。然而，端到端微调旋转优化算法计算成本高且易过拟合。为此，我们提出高效分布感知的旋转校准方法DartQuant，通过约束旋转后激活值的分布来降低旋转优化复杂度，同时减少对任务特定损失的依赖，缓解过拟合风险。此外，引入QR-Orth优化方案，以更高效的方式替代昂贵的交替优化。在多种模型量化实验中，DartQuant表现优异：相较于现有方法，70B模型的旋转优化实现47倍加速和10倍内存节省。首次在单张3090显卡上成功完成70B模型的旋转校准，使大语言模型量化在资源受限环境下成为可能。代码已开源。
原文摘要 · Abstract (English)
Quantization plays a crucial role in accelerating the inference of large-scale models, and rotational matrices have been shown to effectively improve quantization performance by smoothing outliers. However, end-to-end fine-tuning of rotational optimization algorithms incurs high computational costs and is prone to overfitting. To address this challenge, we propose an efficient distribution-aware rotational calibration method, DartQuant, which reduces the complexity of rotational optimization by constraining the distribution of the activations after rotation. This approach also effectively reduces reliance on task-specific losses, thereby mitigating the risk of overfitting. Additionally, we introduce the QR-Orth optimization scheme, which replaces expensive alternating optimization with a more efficient solution. In a variety of model quantization experiments, DartQuant demonstrates superior performance. Compared to existing methods, it achieves 47$\times$ acceleration and 10$\times$ memory savings for rotational optimization on a 70B model. Furthermore, it is the first to successfully complete rotational calibration for a 70B model on a single 3090 GPU, making quantization of large language models feasible in resource-constrained environments. Code is available at https://github.com/CAS-CLab/DartQuant.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。