按顺序剪枝、量化、蒸馏,显著提升模型在边缘设备的推理速度和效率。
Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression
- 采用剪枝-量化-蒸馏的有序流程,以实测延迟为目标优化模型压缩。
- 在CIFAR数据集上实现0.99-1.42毫秒的CPU推理延迟,精度依然保持竞争力。
- 适合对推理速度敏感的边缘部署场景,提供可复现的压缩指导。
现代部署常需在有限的CPU和内存约束下权衡精度与效率,但常见的压缩指标如参数量或浮点运算次数无法可靠预测实际运行时间。特别是非结构化稀疏性虽能减少存储,却因不规则内存访问和稀疏核开销,往往无法加速甚至略微拖慢标准CPU执行。针对这一压缩与加速间的差距,本文研究了一种面向实测延迟的实用有序压缩流程,结合三种常用技术:非结构化剪枝、INT8量化感知训练(QAT)和知识蒸馏(KD)。实验表明,INT8 QAT带来主要的运行时收益,剪枝主要作为容量缩减的预处理步骤,增强后续低精度优化的鲁棒性;而蒸馏最后应用,在已固定的稀疏INT8架构中恢复精度而不改变部署形式。在使用ResNet-18、WRN-28-10和VGG-16-BN三类主干网络的CIFAR-10/100上评估,该有序流程在所有设置下均优于单一技术,实现0.99–1.42毫秒的CPU延迟,同时保持高精度与紧凑检查点。固定20/40/40轮次分配的控制性排序消融实验进一步验证顺序至关重要,所提顺序在所有测试排列中表现最优。总体而言,研究结果为边缘部署提供了简单指导:应基于实测运行时间,在精度-大小-延迟联合空间中评估压缩方案,而非仅依赖代理指标。
原文摘要 · Abstract (English)
Modern deployment often requires trading accuracy for efficiency under tight CPU and memory constraints, yet common compression proxies such as parameter count or FLOPs do not reliably predict wall-clock inference time. In particular, unstructured sparsity can reduce model storage while failing to accelerate (and sometimes slightly slowing down) standard CPU execution due to irregular memory access and sparse kernel overhead. Motivated by this gap between compression and acceleration, we study a practical, ordered pipeline that targets measured latency by combining three widely used techniques: unstructured pruning, INT8 quantization-aware training (QAT), and knowledge distillation (KD). Empirically, INT8 QAT provides the dominant runtime benefit, while pruning mainly acts as a capacity-reduction pre-conditioner that improves the robustness of subsequent low-precision optimization; KD, applied last, recovers accuracy within the already constrained sparse INT8 regime without changing the deployment form. We evaluate on CIFAR-10/100 using three backbones (ResNet-18, WRN-28-10, and VGG-16-BN). Across all settings, the ordered pipeline achieves a stronger accuracy-size-latency frontier than any single technique alone, reaching 0.99-1.42 ms CPU latency with competitive accuracy and compact checkpoints. Controlled ordering ablations with a fixed 20/40/40 epoch allocation further confirm that stage order is consequential, with the proposed ordering generally performing best among the tested permutations. Overall, our results provide a simple guideline for edge deployment: evaluate compression choices in the joint accuracy-size-latency space using measured runtime, rather than proxy metrics alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。