量化让模型对少数群体更不公平，提出混合精度训练缓解问题
Explaining How Quantization Disparately Skews a Model
- 分析量化如何通过权重、激活变化引发群体间性能差异
- 量化后群体准确率下降，梯度范数与海森矩阵特征值异常
- 结合混合精度QAT与加权损失，提升量化模型公平性
后训练量化（PTQ）因其高压缩率和低精度损失被广泛应用。然而我们发现，量化会加剧对少数群体的不公平影响。分析表明，量化在前向与反向传播中引发一系列连锁效应：权重与激活的量化改变导致网络中逻辑值方差降低、损失增大，进而损害群体准确率。进一步研究显示，这些变化影响了群体梯度范数与海森矩阵特征值，揭示了优化状态的失衡。为缓解此问题，我们提出将混合精度量化感知训练（QAT）与数据采样策略及加权损失函数结合，实现更公平的量化神经网络部署。
原文摘要 · Abstract (English)
Post Training Quantization (PTQ) is widely adopted due to its high compression capacity and speed with minimal impact on accuracy. However, we observed that disparate impacts are exacerbated by quantization, especially for minority groups. Our analysis explains that in the course of quantization there is a chain of factors attributed to a disparate impact across groups during forward and backward passes. We explore how the changes in weights and activations induced by quantization cause cascaded impacts in the network, resulting in logits with lower variance, increased loss, and compromised group accuracies. We extend our study to verify the influence of these impacts on group gradient norms and eigenvalues of the Hessian matrix, providing insights into the state of the network from an optimization point of view. To mitigate these effects, we propose integrating mixed precision Quantization Aware Training (QAT) with dataset sampling methods and weighted loss functions, therefore providing fair deployment of quantized neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。