用可学习正则化提升小模型量化压缩效果，兼顾精度与效率。
Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer
- 引入可学习正则器动态调节任务与蒸馏损失权重
- 在低比特量化下实现比现有方法更高的模型精度
- 适合边缘设备部署，尤其对资源受限场景友好
量化感知训练（QAT）结合知识蒸馏（KD）是压缩人工智能模型以适配资源受限硬件的有前景策略。然而，现有QAT-KD方法常因梯度幅值异质性难以平衡任务特定（TS）与蒸馏损失，尤其在低比特量化时更为显著。本文提出游戏正则器（GoR），一种仅含两个可训练参数的可学习正则化方法，通过自适应权重动态平衡TS与KD目标。GoR缓解了监督信号冲突，提升收敛性，并显著增强小量化模型（SQMs）性能。在图像分类、目标检测（OD）及大语言模型（LLM）压缩任务中，GoR持续优于当前最优的QAT-KD方法。在低功耗边缘设备上，其推理速度更快，同时保持全精度准确率。此外，我们提出QAT-EKD-GoR集成蒸馏框架，利用多个异构教师模型。在最佳条件下，该方法甚至超越全精度模型，为实际部署提供稳健解决方案。
原文摘要 · Abstract (English)
Quantization-aware training (QAT) combined with knowledge distillation (KD) is a promising strategy for compressing Artificial Intelligence (AI) models for deployment on resource-constrained hardware. However, existing QAT-KD methods often struggle to balance task-specific (TS) and distillation losses due to heterogeneous gradient magnitudes, especially under low-bit quantization. We propose Game of Regularizer (GoR), a novel learnable regularization method that adaptively balances TS and KD objectives using only two trainable parameters for dynamic loss weighting. GoR reduces conflict between supervision signals, improves convergence, and boosts the performance of small quantized models (SQMs). Experiments on image classification, object detection (OD), and large language model (LLM) compression show that GoR consistently outperforms state-of-the-art QAT-KD methods. On low-power edge devices, it delivers faster inference while maintaining full-precision accuracy. We also introduce QAT-EKD-GoR, an ensemble distillation framework that uses multiple heterogeneous teacher models. Under optimal conditions, the proposed EKD-GoR can outperform full-precision models, providing a robust solution for real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。