arXiv:2601.11663cs.LGcs.AI2026-01

提出激活敏感度统一框架,解释主流量化方法为何有效

Activation Sensitivity as a Unifying Principle for Post-Training Quantization

  • 用梯度加权激活的平方范数定义通道重要性
  • 证明AWQ和GPTQ是不同假设下的敏感度近似
  • 为量化方法设计提供理论依据,适合模型压缩研究者

大型语言模型的后训练量化(PTQ)方法依赖启发式规则,隐式估计哪些权重通道对模型行为影响最大。两种主流范式:如AWQ的激活感知方法关注大激活幅值的通道,而如GPTQ的二阶方法根据输入协方差结构分配量化误差。尽管表现优异,这些方法概念上分散,且不清楚它们实际逼近的底层量是什么。本文通过形式化激活敏感度——即通道扰动对损失的期望影响——建立统一理论框架。利用一阶泰勒展开,我们证明敏感度自然表现为梯度加权激活的平方范数,成为衡量通道重要性的合理指标,同时捕捉激活幅度与下游误差传播。在此框架下,AWQ与GPTQ可被解释为在不同简化假设下恢复敏感度的互补近似。我们分析了敏感度度量的设计空间,关联梯度显著性、费舍尔信息与海森矩阵标准,并阐明其与经典剪枝方法(如最优大脑损伤、最优大脑手术)的关系。本文不提出新量化算法,而是为理解与比较后训练量化方法提供概念基础。

原文摘要 · Abstract (English)

Post-training quantization (PTQ) methods for large language models rely on heuristics that implicitly estimate which weight channels most strongly influence model behavior. Two dominant paradigms have emerged: activation-aware methods such as AWQ prioritize channels with large activation magnitudes, while second-order methods such as GPTQ allocate quantization error according to input covariance structure. Despite strong empirical performance, these approaches remain conceptually fragmented, and it is unclear what underlying quantity they are approximating. In this work, we present a unified theoretical framework for PTQ by formalizing activation sensitivity, defined as the expected impact of channel-wise perturbations on the loss. Using a first-order Taylor expansion, we show that sensitivity naturally arises as the squared norm of gradient-weighted activations, yielding a principled measure of channel importance that captures both activation magnitude and downstream error propagation. Within this framework, AWQ and GPTQ can be interpreted as complementary approximations that recover sensitivity under distinct simplifying assumptions. We analyze the design space of sensitivity metrics, connect gradient-based saliency, Fisher information, and Hessian-based criteria, and clarify their relationships to classical pruning methods such as Optimal Brain Damage and Optimal Brain Surgeon. Rather than proposing a new quantization algorithm, this work provides a conceptual foundation for understanding and comparing post-training quantization methods through the lens of sensitivity.

模型量化敏感度分析理论框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。