通过偏移机制缓解大模型量化中的异常激活，提升低比特推理性能。
OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
- 用Top-1PCA识别激活异常子空间，旋转集中高幅值激活到单通道。
- 将集中通道的幅值转为共享偏移量，降低激活标准差。
- 支持部署友好的统一网格量化，适用于多种大模型架构。
低比特量化被广泛用于加速大语言模型（LLMs）推理，显著降低计算成本与内存占用。然而，激活异常值对有效量化构成重大挑战，常导致性能明显下降。本文提出OffQ，一种通过新型偏移机制缓解激活异常值的方法。具体而言，OffQ首先利用提出的Top-1 PCA在激活中识别低维异常子空间，再通过旋转将高幅值激活集中至1个通道；随后将该通道的幅值转化为共享偏移量，从而减小激活的标准差。此偏移策略使部署友好的统一网格与统一精度量化下实现有效的W4A4KV4 LLM量化。在多种大模型架构和基准上的大量实验表明，OffQ优于现有最先进方法，在保持低比特效率的同时持续提升模型准确率。
原文摘要 · Abstract (English)
Low-bit quantization has been widely adopted to accelerate the inference of large language models (LLMs) by significantly reducing computational cost and memory usage. However, activation outliers pose a major challenge to effective quantization, often leading to notable performance degradation. In this paper, we introduce OffQ, a method designed to mitigate activation outliers in low-bit quantization through a novel offsetting mechanism. Specifically, OffQ first identifies a low-dimensional outlier subspace in the activations using a proposed top-1 PCA, and then concentrates high-magnitude activations into 1 channel via rotation. OffQ then absorbs this concentrated outlier channel by converting its magnitude into a shared offset, thereby reducing the standard deviation of the activations. This offsetting strategy enables effective W4A4KV4 quantization of LLMs using deployment-friendly uniform-grid and uniform-precision quantization. Extensive experiments across diverse LLM architectures and benchmarks demonstrate that OffQ outperforms state-of-the-art baselines, consistently improving model accuracy while preserving low-bit efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。