通过旋转与分组量化协同设计，实现4比特大模型高效推理
GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference
- 提出CoRFiG与HAP机制，解决旋转与分组量化不匹配问题
- 在LLaMA系列模型上达4比特最佳精度，提速3.4倍、能效提升3.6倍
- 适合追求低比特部署的模型压缩与硬件加速研发人员
低比特量化对高效大语言模型推理至关重要，旋转和细粒度分组量化各自展现潜力，但结合时因全局旋转与局部缩放行为不匹配，常导致精度下降或硬件开销增加。我们提出GyRot，一种算法-硬件协同设计的量化框架与加速器。通过引入粗旋转、细分组（CoRFiG）与谐波对齐置换（HAP），实现旋转与分组量化的协同整合，提升可量化性并降低缩放因子精度要求。为减少硬件成本，重构非对称量化并提出零点舍入策略，实现全整数反量化。基于INT4张量处理单元架构实现，GyRot在LLaMA系列模型上达到4比特最优精度，相较基线加速器最高提速3.4倍、能效提升3.6倍。结果验证了其在可扩展、低功耗大模型部署中的实际有效性。
原文摘要 · Abstract (English)
Low-bit quantization is essential for efficient LLM inference, and both rotation and fine-grained group quantization have shown individual promise. However, their combination often leads to accuracy degradation or hardware overhead due to a mismatch between the global nature of rotation and the localized behavior of group scaling. We propose GyRot, a quantization framework and hardware accelerator that bridges this gap through algorithm-hardware co-design. GyRot introduces Coarse Rotation, Fine Grouping (CoRFiG) and Harmonic-Aligned Permutation (HAP) to enable cooperative integration of rotation and group quantization, enhancing quantizability while relaxing scaling factor precision. To further reduce hardware cost, we reformulate asymmetric quantization and introduce a zero-point rounding strategy that enables fully integer dequantization. Implemented on an INT4-based tensor PE architecture, GyRot achieves state-of-the-art 4-bit accuracy across LLaMA-family models, while delivering up to 3.4x speedup and 3.6x energy efficiency over baseline LLM accelerators. These results validate GyRot's practical effectiveness for scalable and energy-efficient LLM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。