让量化教师特征适配学生,实现无标签高效模型压缩。
SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features

- 用学生模型的量化参数重新量化教师特征,消除分布不匹配。
- 在1-2比特下显著超越基线,低比特性能提升最明显。
- 不依赖特定结构,通用性强,适合各类模型与量化场景。
量化感知训练(QAT)可使量化模型在部署时几乎无精度损失。但在实际中,因隐私、版权或成本限制,训练标签常不可用。知识蒸馏(KD)是常见解决方案,但现有结合QAT与KD的方法存在根本缺陷:蒸馏过程中教师与量化学生模型之间的范围不匹配,导致无法消除的残差,形成蒸馏损失的不可降低下界。为此,我们提出SQuaT(Student-Aware Quantized Teacher Features),一种无标签的QAT框架,通过将学生模型的量化参数用于蒸馏阶段对教师特征进行量化,理论上消除了该下界。在多种设置下的全面实验表明,SQuaT持续优于强基线,尤其在极端低比特(如1-和2比特)设置中表现突出。广泛评估显示,本方法不依赖特定架构假设,适用于多样化的模型与量化配置。源代码见https://github.com/lcdbsa522/SQuaT。
原文摘要 · Abstract (English)
Quantization-Aware Training (QAT) enables the deployment of quantized models with minimal accuracy degradation. However, in practical scenarios, training labels are often unavailable due to privacy, copyright, or cost constraints. Knowledge Distillation (KD) is a common approach to address this challenge, but we observe that prior work combining QAT with KD suffers from a fundamental limitation: during distillation, the range mismatch between the teacher and the quantized student model induces an unattainable residual, resulting in an irreducible lower bound on the distillation loss. Motivated by this observation, we propose SQuaT (Student-Aware Quantized Teacher Features), a label-free QAT framework with KD that theoretically eliminates this lower bound by applying the student's quantization parameters to quantize the teacher's features during distillation. Through comprehensive experiments across diverse settings, we demonstrate that SQuaT consistently outperforms strong baselines, with particularly pronounced gains in extreme low-bit (e.g., 1- and 2-bit) settings. Furthermore, extensive evaluations across various model design choices show that our approach does not rely on specific architectural assumptions, making it broadly applicable across diverse architectures and quantization settings. The source code is available at https://github.com/lcdbsa522/SQuaT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。