多教师知识蒸馏框架，自动调参且压缩效率高。
HPM-KD: Hierarchical Progressive Multi-Teacher Framework for Knowledge Distillation and Efficient Model Compression
- 分层渐进式蒸馏，自动构建中间模型链。
- 压缩10至15倍仍保持85%准确率，训练提速30%-40%。
- 适合需要高效模型压缩的工业部署场景。
知识蒸馏（KD）是模型压缩的有力方法，但存在超参数敏感、师生模型容量差距大、多教师协同差及计算资源利用低等挑战。本文提出HPM-KD框架，集成六项协同组件：(i) 基于元学习的自适应配置管理器，消除手动调参；(ii) 自动确定中间模型的渐进蒸馏链；(iii) 动态学习样本级权重的注意力加权多教师集成；(iv) 元学习温度调度器，自适应调整训练中温度；(v) 智能负载均衡的并行处理管道；(vi) 跨实验复用的共享优化内存。在CIFAR-10、CIFAR-100和表格数据集上的实验表明，HPM-KD实现10-15倍压缩，准确率保留达85%，无需人工调参，训练时间减少30%-40%。消融实验验证各组件独立贡献（提升0.10-0.98个百分点）。HPM-KD已开源于DeepBridge库。
原文摘要 · Abstract (English)
Knowledge Distillation (KD) has emerged as a promising technique for model compression but faces critical limitations: (1) sensitivity to hyperparameters requiring extensive manual tuning, (2) capacity gap when distilling from very large teachers to small students, (3) suboptimal coordination in multi-teacher scenarios, and (4) inefficient use of computational resources. We present \textbf{HPM-KD}, a framework that integrates six synergistic components: (i) Adaptive Configuration Manager via meta-learning that eliminates manual hyperparameter tuning, (ii) Progressive Distillation Chain with automatically determined intermediate models, (iii) Attention-Weighted Multi-Teacher Ensemble that learns dynamic per-sample weights, (iv) Meta-Learned Temperature Scheduler that adapts temperature throughout training, (v) Parallel Processing Pipeline with intelligent load balancing, and (vi) Shared Optimization Memory for cross-experiment reuse. Experiments on CIFAR-10, CIFAR-100, and tabular datasets demonstrate that HPM-KD: achieves 10x-15x compression while maintaining 85% accuracy retention, eliminates the need for manual tuning, and reduces training time by 30-40% via parallelization. Ablation studies confirm independent contribution of each component (0.10-0.98 pp). HPM-KD is available as part of the open-source DeepBridge library.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。